AFRL-RH-WP-TR-2 01 0-0081 


BrainPort®  Technology  Tongue  Interface  Characterization 


Wicab,  Inc. 

8476  Greenway  Blvd 
Middleton  Wl  53562 


MARCH  2010 
Final  Report 


Approved  for  public  release;  distribution  is  unlimited. 

See  additional  restrictions  described  on  inside  pages 


AIR  FORCE  RESEARCH  LABORATORY 
711™  HUMAN  PERFORMANCE  WING, 
HUMAN  EFFECTIVENESS  DIRECTORATE, 
WRIGHT-PATTERSON  AIR  FORCE  BASE,  OH  45433 
AIR  FORCE  MATERIEL  COMMAND 
UNITED  STATES  AIR  FORCE 


NOTICE  AND  SIGNATURE  PAGE 


Using  Government  drawings,  specifications,  or  other  data  included  in  this  document  for 

any  purpose  other  than  Government  procurement  does  not  in  any  way  obligate  the  U.S.  Government. 

The  fact  that  the  Government  fonnulated  or  supplied  the  drawings, 

specifications,  or  other  data  does  not  license  the  holder  or  any  other  person  or  corporation; 
or  convey  any  rights  or  permission  to  manufacture,  use,  or  sell  any  patented  invention  that 
may  relate  to  them. 

This  report  was  cleared  for  public  release  by  the  88th  Air  Base  Wing  Public  Affairs  Office  and  is 
available  to  the  general  public,  including  foreign  nationals.  Copies  may  be  obtained  from  the 
Defense  Technical  Information  Center  (DTIC)  (http://www.dtic.mil). 

AFRL-RH-WP-TR-20 10-0081  HAS  BEEN  REVIEWED  AND  IS  APPROVED  FOR  PUBLICATION 
IN  ACCORDANCE  WITH  ASSIGNED  DISTRIBUTION  STATEMENT. 


//SIGNED//  //SIGNED// 

Ed  Eveland,  Work  Unit  Manager  Mark  M.  Hoffman,  Deputy 

Air  Crew  Perfonnance  and  Protection  Branch  Biosciences  and  Protection  Division 

Human  Effectiveness  Directorate 
71 1th  Human  Performance  Wing 
Air  Force  Research  Laboratory 


This  report  is  published  in  the  interest  of  scientific  and  technical  infonnation  exchange,  and  its  publication 
does  not  constitute  the  Government4 s  approval  or  disapproval  of  its  ideas  or  findings. 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the 
data  needed,  and  completing  and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing 
this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202- 
4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently 
valid  OMB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


2.  REPORT  TYPE 


1.  REPORT  DATE  (DD-MM-YYYY) 

10  Mar  2010 


4.  TITLE  AND  SUBTITLE 


Final  Report 


Brain  Port1'  Technology  Tongue  Interface  Characterization 


3.  DATES  COVERED  (From  -  To) 

November  2006  -  January  2010 


5a.  CONTRACT  NUMBER 

FA8650-07-C-7705 


5b.  GRANT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Wicab,  Inc. 

8476  Greenway  Blvd 
Middleton,  WI  53562 


9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Materiel  Command 

Air  Force  Research  Laboratory 

Human  Effectiveness  Directorate 

Biosciences  and  Protection  Division 

Air  Crew  Perfonnance  and  Protection  Branch 

Wright-Patterson  AFB  OH  45433-7028 


12.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 

88ABW/PA  cleared  16  Jun  10;  88ABW-20 10-3321 


14.  ABSTRACT 


5c.  PROGRAM  ELEMENT  NUMBER 

63768E 


5d.  PROJECT  NUMBER 

OS 


5e.  TASK  NUMBER 

MSWP 


5f.  WORK  UNIT  NUMBER 

OSMSWP12 


8.  PERFORMING  ORGANIZATION  REPORT 
NUMBER 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 

711  HPW/RHPA 


11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

AFRL-RH-WP-TR-20 10-0081 


Wicab,  Inc.  has  developed  novel  technology  which  allows  information  from  external  devices  to  be  sensed  by 
humans  via  neuro-stimulation  of  the  tongue. 

This  report  summarizes  a  multi-year  effort  to  empirically  measure  the  electro-stimulation  properties  of  the  tongue, 
including  spatial  and  temporal  discrimination,  contrast  sensitivity,  and  stimulation  optimization  approaches. 


16.  SECURITY  CLASSIFICATION  OF: 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

U 

U 

U 

17.  LIMITATION  18.  NUMBER  19a.  NAME  OF  RESPONSIBLE  PERSON 

OF  ABSTRACT  OF  PAGES  Ed  gyeland 

19b.  TELEPHONE  NUMBER  (include  area 

SAR  54  "%A 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  239.18 


THIS  PAGE  IS  INTENTIONALLY  LEFT  BLANK 


TABLE  OF  CONTENTS 


1  PROJECT  SUMMARY . 1 

1.1  Significant  Findings . 2 

1 .2  Program  Information . 5 

2  STUDIES  AND  RESULTS . 6 

2.1  Two-Point  Discrimination . 6 

2.2  Temporal  GAP  Discrimination . 8 

2.3  T ongue  Mapping  &  Spatial  Summation . 10 

2.4  Contrast  Sensitivity . 11 

2.5  Aggregate  Channel  Capacity . 15 

2.6  Enhanced  Percept  -  Rifleman  Training . 20 

3  HARDWARE  AND  SOFTWARE  DEVELOPMENTS . 24 

3.1  High  Density  Array  Control  System . 24 

3.2  Hardware . 25 

3.3  Software . 30 

3.4  Stimulation  Platform . 30 

3.4.1  Virtual  Machine  Concept . 30 

3.4.2  HD  Controller . 32 

3.4.3  WG-C200  . 33 

3.5  Software  Modules . 33 

3.6  Workstation  PC  Software  Modules . 33 

3.6.1  Experiment  Controller . 33 

3.6.2  Utilizing  the  VM  to  Present  T rials . 34 

3.6.3  HD  Diagnostic  Utility . 35 

3.6.4  HD  Controller  Library  and  the  WGIO  Protocol . 35 

3.7  HD  Controller  Board  Modules . 36 

3.7.1  Gumstix  Platform . 36 

3.7.2  Hardware  Interconnects . 37 

3.7.3  Board-to-Board  Waveform  Synchronization . 38 

3.7.4  Master  Clock/PWM  Signal  Output . 38 

3.7.5  Waveform  SRAM . 38 

3.7.6  Power  Supply  Control . 39 

3.7.7  DIP  Switches/LED  Indicators . 40 

3.7.8  FPGA . 40 

3.8  JTAG  Programming . 41 

3.9  Data  Formats . 42 

3.10  Large  Area  Array  and  VI 00  Software  Support . 42 

3.11  X16BPV-V1 00  Interface . 42 

3.12  TUNS  vSight  Feature  Packs . 43 

4  WAVEFORM  OPTIMIZATION . 44 

5  PUBLICATIONS . 45 

6  REFERENCES . 45 

Appendix  A  -  Virtual  Machine  Instruction  Set . 46 

Appendix  B  -  Virtual  Machine  State  Definition . 48 

Appendix  C  -  WGIO  Packet  Definitions . 49 

LIST  OF  ABBREVIATIONS . 50 

iii 


LIST  OF  FIGURES 


Figure  1.  Original  Electrode  Array  (10  electrodes/row)  and  High  Density  Strip  Array  (100 

electrodes)  . 3 

Figure  2.  Modified  Mock  Weapon  for  Rifleman  Training  . 4 

Figure  3.  Two  Point  Discrimination  Across  Participants  . 8 

Figure  4.  Gap  Detection  Across  Participants  . 9 

Figure  5.  Tongue  Sensitivity  Maps  . 10 

Figure  6.  Spatial  Summation . 1 1 

Figure  7.  Contrast  Sensitivity  Experiment  Design  . 12 

Figure  8.  Main  Effect  -  Gap  Spacing  . 13 

Figure  9.  Main  Effect  -  Contrast  Voltage  . 13 

Figure  10.  Contrast  Sensitivity  Chart . 14 

Figure  11.  Vernier  Test  . 16 

Figure  12.  Flicker  Fusion . 19 

Figure  13.  Training  Group  Performance . 23 

Figure  14.  High  Density  Array  Control  System  Architecture  . 24 

Figure  15.  Core  Hardware  Architecture . 25 

Figure  16.  Complete  HD  Control  Board . 26 

Figure  17.  Control  Board  Insertion . 26 

Figure  18.  Populated  Instrumentation  Rack . 27 

Figure  19.  High  Density  Array  -  100  element  strip,  compared  to  original  10x10  array  . 28 

Figure  20.  Strip  Array  Stack-up . 29 

Figure  21.  Stackup  and  single  strip  array . 30 

Figure  22.  Stimulus  Program  WorkFlow . 31 

Figure  23.  HD  Server/Client  Communication  . 34 

Figure  24.  SRAM  Memory  Map . 39 


IV 


1  PROJECT  SUMMARY 


Wicab,  Inc.  has  developed  novel  technology  which  allows  information  from  external  devices  to 
be  sensed  by  humans  via  neuro-stimulation  of  the  tongue.  Applications  are  numerous  and  include 
the  Brain  Port '  balance  device  to  assist  patients  with  vestibular  deficits,  the  BrainPort  vision 
device,  a  sensory  augmentation  device  for  the  blind,  and  the  BrainPort  Underwater  Sensory 
Substitution  System  which  provides  navigational  cues  to  military  divers.  Current  devices  present 
data  to  the  tongue  using  low  density  electrode  arrays  (100-625  elements,  ~6.45cm2).  Present 
and  future  applications  may  benefit  from  array  densities  and  electro-stimulation  waveform 
patterns  that  match  the  spatial/temporal  resolution  of  the  tongue.  Therefore,  characterization  of 
this  resolution  is  the  key  to  full  exploitation  of  the  tongue  sensory  channel. 

This  DARPA-funded  research  program  is  a  multi-year  effort  to  empirically  measure  the  electro¬ 
stimulation  properties  of  the  tongue.  Advanced  hardware  devices  and  associated  software  were 
developed  to  support  experimental  protocols.  Under  this  Program  approximately  60  subjects 
and  more  than  200  subject  hours  were  involved  in  the  following  experiments: 

•  Two-point  discrimination 

•  Temporal  gap  detection 

•  Tongue  mapping 

•  Spatial  summation 

•  Contrast  sensitivity 

•  Neuro-stimulation  optimization 

•  Percept  Training  -  Sharpshooter  simulation 

All  experiments  involving  human  subjects  were  conducted  under  an  IRB  approved  protocol. 

The  Program1  s  objective  was  to  provide  empirical  evidence  concerning  the  psychophysical  limits 
of  the  tongue  in  order  to  characterize  its  information  capacity. 

Prior  to  this  program- 

■  Little  was  known  regarding  the  psychophysical  limits  of  tongue  electro -stimulation 

■  Minimum  electrode  size  and  aggregate  electrode  density  limits  for  tongue  stimulation 
were  guesses,  at  best 

■  It  was  unknown  whether  electrode  arrays  with  sub-millimeter  diameter  electrodes  and 
associated  circuitry  could  be  could  be  fabricated  and  safely  driven  such  that  sufficient 
charge  reaches  the  tongue‘s  receptors  to  allow  detection  (i.e.  would  the  minute  charge 
find  return  paths  before  reaching  the  depth  of  the  receptors) 

■  It  was  unknown  whether  a  human  subject  could  detect  electrical  stimulation  pulses  on  the 
tongue  presented  via  electrodes  in  the  sub-millimeter  diameter  range 


The  results  of  this  program  clearly  demonstrate  that  the  human  tongue  is  capable  of 
temporal/spatial  detection  of  electrical  signals  at  least  equal  to,  or  better  than,  other  electro- 
tactilely  sensitive  locations  (fingertips,  for  example)  and  that  electro-stimulating  arrays  with 
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electrode  diameters  and  spacing  of  less  than  250  microns  can  present  useful  infonnation.  The 
results  also  demonstrate  that  there  are  individual  variations  in  task  perfonnance.  Test  subjects 
were  not  chosen  for  their  tongue  perception  ability,  so  — supr-perfonners”  which  would  improve 
the  threshold  limit  may  have  not  have  been  tested. 

While  this  suite  of  studies  provides  empirical  data  regarding  the  psychophysical  characteristics 
of  the  tongue,  they  should  not  be  interpreted  as  the  absolute  limits.  The  results  from  these 
experiments  begin  to  characterize  the  infonnation  channel,  but  they  do  not  represent  the  true 
limits  of  the  channel.  In  order  to  assess  the  absolute  limit,  four  experimental  components  need  to 
be  optimized:  subjects,  training,  application,  and  technology.  This  was  beyond  the  scope  of  the 
current  program. 


1,1  Significant  Findings 

•  Electro-tactile  psychophysical  characteristics  of  the  tongue 

Two  Point  Spatial  Discrimination  is  0.75mm  on  average,  with  some  individuals  able  to 
detect  0.25mm  or  better.  This  implies  that  arrays  on  the  order  of  0.125mm  center-center 
electrode  spacing  (or  better)  may  be  useful  for  high  performing  individuals. 

Temporal  Discrimination  is  on  average  75ms,  with  some  individuals  perfonning  at  50ms 
or  better  gap  detection.  This  implies  that  array  frame  rates  of  20  Hz  or  faster  should  be 
useful  for  high  performing  individuals. 

Test  data  suggests  that  tongue  sensitivity  is  a  function  of  stimulation  location  as  well  as 
number  of  stimulating  tactors.  In  addition,  the  dynamic  range  in  voltage  (stimulation 
intensity)  is  a  function  of  the  number  of  stimulating  tactors.  This  data  may  have 
implications  on  the  electro-mechanical  design  of  high  density  arrays,  as  well  as  the 
stimulation  waveform  and  power  requirements. 

Preliminary  data  from  static  2-D  grating  tests  indicates  that  average  spatial 
discrimination  is  0.6mm  (compared  to  0.75mm  for  2pt  tests).  This  implies  that  there  may 
be  strategies  to  further  improve  the  average  spatial  discrimination  ability. 

Spatial  acuity  and  temporal  acuity  appear  to  differ  within  individuals  -  high  performers  in 
one  modality  are  not  necessarily  high  performers  in  another.  This  presents  opportunities 
to  train  individuals  whose  physiology  allows  peak  perfonnance  across  modalities,  to 
tailor  information  to  individual  perfonnance,  and  to  tailor  anays  for  specific  applications. 

Finally,  data  from  this  study  confirms  that  electro-neurostimulation  of  tongue  is  on  par 
with  other  stimulation  modalities:  mechanical  grating  ~0. 500mm  (Van  Boven,  1994)  and 
vibro-tactile  at~20Hz  (Ezawa,  1988). 

Note  that  for  these  studies,  subjects  did  not  receive  extensive  practice  with  the  stimuli, 
which  may  extend  the  psychophysical  limits.  Also,  researchers  found  that  task  demands 
affect  the  reported  threshold.  For  example,  two  point  discrimination  was  ~0.75mm  but 
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spatial  gratings  threshold  was  closer  to  ~0.5mm.  Therefore,  to  define  the  absolute 
threshold,  either  the  task  needs  to  be  fixed  to  a  task  of  interest  or  there  needs  to  be 
converging  evidence  from  a  variety  of  experiments  testing  similar  skills.  Finally,  subject 
performance  is  only  as  precise  as  the  equipment  being  used.  To  report  a  definite 
threshold,  additional  instrumentation  suitable  for  this  unique  environment  is  necessary. 


•  High  Density  Array  Design  and  Experiment  Control 

Experimental  data  indicates  0.125mm  electrode  spacing  (200/inch)  should  match  the 
tongue‘s  spatial  acuity.  Using  a  simple  fabrication  technique,  arrays  with  0.152mm 
spacing  (Figure  1)  were  assembled  (densities  beyond  this  level  requires  a  significant 
technology  investment).  These  near-optimal  arrays  allow  advanced  testing  of  dynamic 
information  on  the  tongue. 


Figure  1.  Original  Electrode  Array  (10  electrodes/row)  and  High  Density  Strip  Array  (100 

electrodes) 

An  electrical  control  system  was  designed  and  implemented  which  allows  concurrent 
control  of  all  electrodes  in  an  array.  This  system  manages  the  safe  delivery  of 
stimulation  to  any  number  of  simultaneous  electrodes,  as  well  as  manipulation  of  the 
return  path  geometry.  In  addition,  stimulation  waveform  patterns  can  be  defined  for 
evaluation  of  wavefonn  on  spatial  and  temporal  detection  properties.  The  system  is 
designed  to  manage  up  to  27,000  electrodes,  although  for  practical  testing,  we  have 
limited  the  current  implementation  to  2000  element  arrays. 

An  experimental  control  software  application  was  developed  to  allow  researchers  to 
define  experiments  using  any  configuration  of  electrode  geometry  and  stimulation 
wavefonn  patterns.  Experiment  sessions  are  defined  and  executed  to  present  stimulus 
information  to  subjects  and  record  their  responses.  All  data  is  archived  for  off-line 
analysis. 

•  Neuro-stimulation  Optimization 

Hardware  and  software  were  developed  to  allow  exploration  of  pulse  frequency  and 
affects  on  spatial  summation  and  discrimination.  Data  from  this  study  supports  the 
conclusion  that  dynamic  range  is  clearly  affected  by  neighboring  stimulation  pulses. 

•  Stimulation  Detection  Strategies 
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Preliminary  exploration  of  this  area  indicated  that  individual  instruction  with  an 
experienced  trainer  has  a  more  significant  affect  on  perceptual  performance  than  does  any 
specific  individual  detection  strategy.  That  is,  individuals  could  be  trained  to  achieve  the 
same  level  of  performance  for  a  given  task  regardless  of  how  they  used  their  tongue  to 
feel  the  stimulation.  Effort  under  this  task  was  combined  with  the  =Rihanced  Percept4 
task  and  others,  to  focus  on  instruction  and  training  development. 

•  Enhanced  Percept  -  Rifleman  Training 

Subjects  were  trained  to  hold  a  rifle  (Figure  2)  with  0  degrees  of  cant  and  were  either 
given  verbal  training  or  automated  training  with  feedback  from  an  integrated  BrainPort- 
based  tongue  display.  Both  groups  achieved  skills  at  holding  the  rifle  in  a  stationary 
vertical  orientation  and  performance  improved  with  training.  There  is  no  clear  difference 
between  subjects  trained  by  human  interaction  or  those  trained  exclusively  with  electro- 
tactual  stimulation. 

These  results  suggest  that  providing  electro-tactile  information  to  the  tongue  is  an 
effective  alternative  method  for  providing  rifle  position  training  to  a  novice.  As  a  result, 
training  techniques  could  be  modified  to  incorporate  aspects  of  automated  training,  thus 
directing  human  interaction  coaching  to  other  aspects  of  rifle  skill  acquisition. 


Figure  2.  Modified  Mock  Weapon  for  Rifleman  Training 
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2  STUDIES  AND  RESULTS 


2.1  Two-Point  Discrimination 

Two-point  discrimination  was  perfonned  using  a  modified  BrainPort  Balance  Device  C200  (BBD- 
C200).  Nine  linear  arrays  with  ten  electrodes  each  were  produced  with  industry  standard  printed 
wiring  board  technology.  The  individual  electrodes  ranged  in  diameter  from  1.5mm  to  0.169mm  and 
were  spaced  (center  to  center)  at  a  distance  of  1.5  times  the  electrode  diameter.  The  smallest  electrode 
size  and  spacing  represents  the  highest  resolution  readily  available  from  commercial  printed  wiring 
board  vendors.  Sufficient  arrays  were  produced  so  that  each  test  subject  used  a  new  array,  minimizing 
the  effects  of  electrode  corrosion  and  resultant  changes  in  impedance. 

The  stimulus  generation  circuit  employed  in  the  BBD-C200  consists  of  a  ColdFire  32-bit 
microcontroller,  Texas  Instruments  digital  to  analog  converter  (TLV5630),  Burr-Brown  operational 
amplifier  (OPA4132),  and  a  hank  of  Analog  Devices  demultiplexers  (ADG408BRU)  used  for  routing 
the  stimulus  signal  to  any  one  of  one  hundred  electrode  addresses.  Each  output  of  the  demultiplexers 
are  connected  to  an  AC  coupling  capacitor  (0.1  pF  ceramic)  with  the  opposite  side  of  that  capacitor 
connected  to  the  tongue  placed  electrode. 

Internal  pilot  studies  suggested  that  some  subjects  could  discriminate  two  stimuli  spaced  0.5mm  apart. 
As  a  result,  we  selected  a  linear  electrode  array  with  10  electrodes,  each  169  micrometers  (p)  diameter 
and  spaced  254p  apart  (center  to  center)  for  the  formal  two  point  discrimination  experiment.  This 
array  enabled  conditions  that  could  both  exceed  perfonnance  and  also  be  within  the  range  of 
successful  perfonnance.  Each  active  electrode  presented  a  continuously  repeating  stimulation 
waveform.  The  waveform  scheme  consisted  of  six  double  stimulating  pulses  (2 1 .4psec  each  separated 
by  5psec)  repeating  at  369Hz.  After  every  6  pulses,  2  pulses  were  presented  at  0V  (total  period  of 
waveform  was  2 1 .68ms).  When  two  electrodes  were  presented,  the  first  electrode  fired  5 1 .6psec  after 
the  second  electrode.  Pulse  amplitude  was  fixed  per  subject.  Each  subject  set  their  own  comfortable 
working  amplitude  up  to  24.5V  prior  to  the  experiment,  providing  comfortable  stimulation.  This 
waveform  scheme  was  selected  because  sensation  was  perceived  as  a  comfortable  and  continuous 
stimulation. 

Sixteen  subjects  (10  men;  6  women)  participated,  ranging  in  age  from  18-39  years  (mean  24.5yrs). 
Experiment  duration  was  approximately  two  hours.  Subjects  were  recruited  from  the  Madison,  WI 
area.  All  subjects  gave  infonned  consent  with  a  study  protocol  approved  by  the  New  England  IRB. 
Pulse  amplitude  (in  volts)  was  detennined  individually  before  running  the  two  point  discrimination 
experiment.  Subjects  were  presented  with  trials  containing  one  or  two  electrodes  firing  on  an  array. 
They  were  instructed  to  manipulate  a  hand  held  slider  control,  allowing  each  subject  to  explore  and 
adjust  the  stimulation  to  a  comfortable  working  level.  Once  a  comfortable  level  was  reached,  subjects 
indicated  their  response  by  pressing  a  button  on  the  hand-held  control.  -W orking  level”  was  defined 
as  a  stimulation  level  that  was  strong  and  comfortable  for  at  least  five  seconds.  Intensity  values 
(voltage)  were  gathered  for  one  and  two  contiguous  firing  electrodes.  Each  measurement  was  repeated 
several  times  across  multiple  electrode  locations.  For  each  subject,  the  mean  -working  level”  for  one 
and  two  contiguous  points  was  used  as  the  fixed  voltage  intensity  for  the  two  point  discrimination  test. 
These  values  ranged  from  18-25V  across  participants  (mean  21.46V). 
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Trials  were  presented  in  five  experimental  runs,  blocked  by  the  separation  between  two  electrodes. 
These  conditions  were:  0.254mm  (no  gap),  0.580mm  (1  electrode  gap),  0.762mm  (2  electrode  gap), 
1.016mm  (3  electrode  gap)  and  1.261mm  (4  electrode  gap).  In  each  block,  half  of  the  trials  contained 
only  one  stimulating  electrode,  while  the  other  half  contained  two  firing  electrodes  of  the  fixed 
condition  distance.  The  order  of  the  five  experimental  blocks  was  pseudo-randomized  for  each 
subject. 

Subjects  were  instructed  to  press  one  button  on  the  hand-held  control  if  they  felt  one  stimulus  on  their 
tongue  and  a  different  button  if  they  felt  two  stimuli  on  their  tongue.  They  were  encouraged  to  use  the 
tip  of  their  tongue  (or  any  other  part  of  the  tongue)  to  search  for  and/or  to  explore  the  stimulation  in 
order  to  respond  maximally.  There  was  no  response  time  limit.  In  this  way,  subjects  could  explore  the 
stimulus  with  whatever  part  of  the  tongue  they  felt  gave  them  the  best  information. 

To  familiarize  subjects  with  electro  tactile  stimulation  and  the  procedures,  subjects  were  given  5-10 
minutes  of  interactive  practice  trials.  Participants  sampled  a  few  trials  of  each  of  the  conditions, 
becoming  acquainted  with  the  trial  types.  During  this  practice  phase,  the  researcher  labeled  the 
conditions  as  being  truly  two  points  or  one.  Subjects  were  told  there  may  be  trials  where  they  may  not 
be  able  to  confidently  give  a  correct  response.  They  were  reassured  that  this  was  necessary  in  order  to 
get  to  the  limit  of  spatial  resolution  on  the  tongue.  During  experimental  testing,  perfonnance  feedback 
was  not  provided. 

The  primary  endpoint  will  be  a  threshold  defined  by  a  group  d-prime  greater  than  or  equal  to  one  (by 
subject  and  by  group),  indicating  subjects  are  sensitive  to  the  presence  of  two  categories  (CBASEE 
1985).  Calculating  d-prime  (d‘),  a  measure  of  sensitivity  in  a  discrimination  task,  is  computed  by  the 
standardized  difference  between  the  false  alarm  rate  and  the  hit  rate:  d‘=z(H)-z(FA)  (Macmillan  & 
Creehnan  2005).  The  hit  rate  is  the  proportion  of  correctly  identified  trials  of  two  points,  when  two 
points  were  actually  stimulated.  The  false  alarm  rate  is  the  proportion  of  incorrect  trials  where  the 
subject  reported  two  points,  but  only  one  point  was  actually  stimulated.  The  numeric  output  of  d- 
prime  represents  the  perceived  distance  between  the  two  categories.  A  d-prime  near  zero  represents 
chance  discrimination,  where  subjects  are  insensitive  to  the  condition  categories.  When  d-prime  is 
greater  than  one,  at  least  one  standard  deviation  separates  the  two  response  categories  indicating  that 
subjects  are  sensitive  to  and  correctly  responding  to  the  condition  categories. 

As  a  secondary  endpoint,  individual  d-prime  measures  by  condition  were  statistically  explored  post- 
hoc  using  a  one  way  ANOVA.  Individual  differences  were  further  explored  by  computing  the 
percentage  of  subjects  whose  individual  performance  exceeded  d-prime  discrimination  greater  than 
one.  Figure  3  illustrates  performance  across  participants. 
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Two  Point  Discrimination 


Figure  3.  Two  Point  Discrimination  Across  Participants 
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The  line  graph  (left  y-axis)  represents  the  group‘s  mean  d-prime  perfonnance  across  the  five  electrode 
spacing  conditions.  As  a  group,  perfonnance  exceeded  d-prime>l  when  electrodes  were  spaced 
0.762mm  (2  electrode  gap).  A  one  way  ANOVA  revealed  a  significant  main  effect  for  electrode 
spacing  (F(4,60)=4.466,  p<.005).  As  the  electrode  spacing  increased,  performance  improved.  The 
line  graph  (right  axis)  illustrates  the  percentage  of  subjects  in  each  condition  who  individually 
exceeded  threshold.  For  example,  12.5%  of  subjects  scored  above  threshold  in  the  most  closely 
spaced  254u  condition.  So  while  the  group  average  threshold  is  0.762mm,  there  are  individuals  who 
have  better  than  average  discrimination  abilities.  This  may  suggest  that  an  individual^  actual  limit 
may  be  better  than  0.254mm  spacing. 

2.2  Temporal  GAP  Discrimination 

Gap  discrimination  was  perfonned  using  a  modified  BrainPort  Balance  Device  C200  (BBD-C200)  as 
described  above.  Internal  pilot  studies  suggested  temporal  gaps  with  the  smallest  array  (0.254mm 
spacing)  were  difficult  to  discriminate.  Using  a  linear  array  with  10  electrodes  sized  0.667mm  spaced 
1.0mm  apart,  internal  subjects  were  able  to  feel  temporal  gaps  of  50-100ms  in  duration.  With  this 
array,  conditions  were  generated  that  could  both  exceed  performance  and  also  be  within  the  range  of 
successful  perfonnance.  All  gap  discrimination  conditions  use  a  continuous  repeating  pulse  scheme 
presenting  1  pulse  for  25  ps  wide  every  1 25ps  (frame)  to  each  electrode  with  a  programmable  -gap”  of 
no  stimulation  on  one  of  the  electrodes.  For  the  -gap”  condition,  the  electrode  pulses  for  a  total  of 
500ms  (four  frames)  followed  by  an  -off  gap”  of  33-250ms. 

Sixteen  subjects  (10  men;  6  women)  participated,  ranging  in  age  from  18-39  years  (mean  24.5yrs). 
Experiment  duration  was  approximately  two  hours.  Subjects  were  recruited  from  the  Madison,  WI 
area.  All  subjects  gave  infonned  consent  with  a  study  protocol  approved  by  the  New  England  IRB. 
Some,  but  not  all,  subjects  participated  in  the  2-pt  Discrimination  Test.  Pulse  amplitude  (in  volts)  was 
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determined  individually  before  running  the  gap  discrimination  experiment.  Subjects  were  presented 
with  trials  containing  electrodes  firing  on  the  array  with  variable  gaps  in  the  continuous  stimulation. 
They  were  instructed  to  manipulate  a  hand  held  slider  control,  allowing  each  subject  to  explore  and 
adjust  the  stimulation  to  a  comfortable  working  level.  Once  a  comfortable  level  was  reached,  subjects 
indicated  their  response,  using  either  a  thumb  or  finger  to  press  a  button  on  the  hand-held  control. 

-W orking  level”  was  defined  as  a  stimulation  level  that  was  strong  and  comfortable  for  at  least  five 
seconds.  Intensity  values  (voltage)  were  repeated  several  times  and  averaged  across  conditions, 
creating  a  mean  ^working  level”  as  the  fixed  voltage  intensity  for  the  gap  discrimination  test. 

This  experiment  consisted  of  five  blocks  of  experimental  runs  containing  twenty  Two -Alternative 
Forced  Choice  (2AFC)  trials  for  a  total  of  100  trials.  Each  trial  contained  two  stimulating  electrodes, 
one  on  the  right  and  one  on  the  left,  spaced  four  electrodes  apart.  For  every  trial,  one  electrode  was 
continuously  firing  (as  defined  above),  while  the  other  electrode  had  a  temporal  gap.  The  targeted  gap 
stimulus  was  presented  -en”  for  500ms  followed  by  one  of  five  -off’  gaps:  33ms,  50ms,  75ms,  100ms 
and  250ms.  This  target  stimulus  cycled  through  -on”  and  -off’  continuously  while  the  other  electrode 
presented  a  continuous  wavefonn.  The  two  stimuli  were  presented  until  the  participant  indicated 
which  stimulating  electrode  had  the  temporal  gap. 

Subjects  were  instructed  to  press  the  left  button  if  they  felt  the  stimulus  had  a  temporal  gap  on  the  left, 
else,  press  the  right  button  if  they  felt  it  on  the  right.  Half  the  correct  responses  for  the  trials  for  each 
were  presented  on  the  right  and  left  respectively.  Again,  the  participant  was  allowed  to  take  as  much 
time  as  needed  and  a  free  moving  tongue  was  allowed.  In  this  way,  subjects  could  explore  the 
stimulus  with  whatever  part  of  the  tongue  they  felt  gave  them  the  best  information.  The  order  of  the 
five  blocks  was  pseudo-randomized  for  each  subject  and  there  was  no  performance  feedback  given. 
Two  point  discrimination  performances  translated  into  a  percent  correct  score  for  each  temporal  gap 
and  subject.  Effective  discrimination  threshold  was  defined  as  when  the  percent  correct  was  greater 
than  75%.  Figure  4  illustrates  performance  across  participants. 

Flicker  Fusion 


n=16  p<0.0001 
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The  line  graph  (left  y-axis)  represents  the  group‘s  mean  d-prime  perfonnance  across  the  five  temporal 
gap  conditions.  As  a  group,  performance  exceeded  d-prime>l  when  electrodes  had  a  gap  of  at  least 
75ms.  A  one  way  ANOVA  revealed  a  significant  main  effect  for  electrode  spacing  (F(4,60)=6.239, 
p<.005).  As  the  temporal  gap  increased,  performance  improved.  The  dotted  line  illustrates  the 
percentage  of  subjects  in  each  condition  who  individually  exceeded  threshold.  While  the  group 
average  threshold  is  75ms  for  gap  detection,  there  are  individuals  who  have  better  than  average 
discrimination  abilities.  This  may  suggest  that  an  individual^  actual  limit  may  be  better  than  50ms. 

2.3  Tongue  Mapping  &  Spatial  Summation 

A  review  and  analysis  of  the  tongue  mapping  and  spatial  summation  data  using  the  25x25  tongue  array 
suggests  tongue  sensitivity  is  a  function  of  stimulation  location  as  well  as  number  of  stimulating 
electrodes.  A  representative  three-dimensional  tongue  sensitivity  map  is  included  ( Figure  5).  The 
front  center  of  the  tongue  has  the  greatest  sensitivity,  as  indicated  by  the  lowest  voltage  region.  As 
stimulation  is  placed  near  the  back  of  the  tongue,  the  mean  sensitivity  is  decreased.  When  many 
electrodes  are  stimulated,  less  voltage  is  required  for  threshold  and  working  levels.  When  one 
electrode  is  stimulated,  more  voltage  is  required  for  threshold  and  working  levels.  The  spatial 
summation  graph,  Figure  6,  illustrates  that  sensitivity  is  a  function  of  the  size  of  the  stimulating  area 
on  the  tongue.  In  addition,  the  graph  illustrates  the  dynamic  range  in  voltage  as  a  function  of  the 
number  of  stimulating  electrodes,  where  there  is  a  greater  range  when  one  electrode  is  firing  as 
compared  to  a  smaller  range  when  nine  electrodes  are  firing. 


Mean  Working  Level 


Figure  5.  Tongue  Sensitivity  Maps 
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Spatial  Summation 


One  Two  Four  Nine 


Size  Tactors 


Figure  6.  Spatial  Summation 


2.4  Contrast  Sensitivity 

Using  the  20x20  portion  of  the  High  Density  TUNS  array  (76pm  diameter  electrode,  152pm  center-to- 
center  spacing),  contrast  sensitivity  was  measured  in  five  participants.  Twenty-five  experimental 
blocks  included  five  trials  each  of  five  spatial  line  gratings  presented  at  five  different  contrast  levels: 


5-gap  (760pm) 
4-gap  (608pm) 
3-gap  (456pm) 
2-gap  (304pm) 
1-gap  (152pm) 


100%  Working  Level  Voltage 
80%  Working  Level  Voltage 
60%  Working  Level  Voltage 
40%  Working  Level  Voltage 
20%  Working  Level  Voltage 


Figure  7  illustrates  the  general  experimental  design  (not  to  scale). 
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Spatial  Resolution 


Figure  7.  Contrast  Sensitivity  Experiment  Design 


Each  experimental  block  consists  of  15  randomly  presented  grating  orientations:  5  horizontal,  5 
vertical  or  5  neither  (diagonal).  Participants  were  required  to  make  a  Three-Alternative  Forced  Choice 
indicating  the  direction  of  the  presented  grating.  Perceptual  threshold  is  reached  in  conditions  where 
percent  correct  exceeds  66.67%. 

A  two-way  ANOVA  (gap-spacing  x  contrast)  was  perfonned,  resulting  in  a  significant  main  effect  of 
gap  spacing,  Figure  8,  (F(4,16)=6.044,  p<.004),  a  significant  main  effect  of  contrast  level,  Figure  9, 
(F(4,16)=5.406,  p<.006),  with  no  significant  interaction. 
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Percent  Correct  Percent  Correct 


Main  Effect  of  Gap  Spacing* 

n=5  p<.004 


760|jm  608|jm  456|jm  304|jm  152|jm 

Gap  Spacing 

Figure  8.  Main  Effect  -  Gap  Spacing 

Main  Effect  of  Contrast  Voltage* 

n=5  p<.006 


Percent  Voltage  from  Working  Level 
Figure  9.  Main  Effect  -  Contrast  Voltage 
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Group  percent  correct  values  for  each  experimental  condition  are  shown  in  the  following  table. 
Conditions  that  meet  exceed  threshold  are  highlighted  in  yellow. 


20% 

760pm 

74.67 

608pm 

57.33 

456pm 

54.67 

304pm 

37.33 

152pm 

40 

40% 

84 

74.67 

57.33 

37.33 

53.33 

60% 

84 

64 

68 

38.67 

58.67 

80% 

82.67 

72 

66.67 

40 

44 

100% 

65.33 

68 

69.33 

42.67 

36.67 

Using  this  data,  a  graphical  — Corrhst  Sensitivity”  chart  is  included,  Figure  10,  (not  to  scale),  with 
conditions  that  meet  or  exceed  threshold  are  highlighted  in  yellow.  Two  conditions,  colored  in  light 
green  (100%/760um,  60%,608um),  are  close  to  threshold  and  one  would  assume  with  more  subjects 
would  be  included  as  above  threshold. 
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Figure  10.  Contrast  Sensitivity  Chart 
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2.5  Aggregate  Channel  Capacity 

Upon  building  two-dimensional  high  density  arrays,  we  explored  and  evaluated  the  tongue‘s  capacity 
to  interpret  information  with  both  spatial  and  temporal  components.  One  specific  goal  was  to  test 
motion  perception  on  the  tongue.  Exploration  included  detennining  interaction  between  stimulation 
size,  movement  velocity  and  frame  rate  on  perceptual  discrimination.  In  addition,  we  explored  the 
ability  of  the  tongue  to  detect  linear  discontinuities  (Vernier  Acuity)  to  discriminate  small  features. 

Motion  Perception 

Motion  perception  on  the  tongue  was  initially  explored  using  our  25x25  vision  arrays  (1mm  center-to- 
center  spacing),  with  points  of  varying  sizes.  Regardless  of  size,  directional  information  was  easy  to 
perceive.  Moreover,  we  could  not  find  a  combination  of  size/speed  that  we  felt  was  challenging 
enough  to  explore  the  limits  of  this  perception.  We  next  explored  this  same  task  using  the  high- 
density  20x20  array,  which  notably  covers  a  much  smaller  portion  of  the  tongue.  As  a  result,  we  found 
that  moving  balls  across  this  small  area  became  too  difficult  to  perceive  the  direction  of  motion.  Most 
users  were  able  to  perceive  orientation,  but  the  motion  did  not  cover  enough  space  to  fonn  an  accurate 
perception  of  direction.  Similar  issues  occurred  when  exploring  lines  and  gratings  with  motion.  Due 
to  the  limited  -workspace”  of  the  20x20  array,  moving  lines  and  bars  appeared  to  fluctuate  or  were 
perceived  as  a  repeating  pulse  on  the  user‘s  tongue  rather  than  the  intended  fluid  movement. 


Applied  Spatial  Resolution:  Vernier  Acuity 

Six  subjects  participated  in  this  experiment  based  upon  the  classic  Vernier  visual  acuity  test,  using  the 
high-density  20x20  array,  where  theoretically  the  spacing  between  individual  electrodes  exceeds  the 
previously  found  spatial  resolution  threshold.  Horizontal  lines  were  presented  and  subjects  were  asked 
whether  they  felt  a  straight  continuous  horizontal  line,  or  one  with  a  discontinuity.  The  experiment 
consisted  of  five  blocks  of  randomly  presented  trials  ranging  from  1-5  electrode  offsets  as  compared  to 
a  continuous  straight  line: 

1 .  6  straight  lines  versus  6  lines  with  one  electrode  shift  off 

2.  6  straight  lines  versus  6  lines  with  two  electrode  shift  off 

3.  6  straight  lines  versus  6  lines  with  three  electrode  shift  off 

4.  6  straight  lines  versus  6  lines  with  four  electrode  shift  off 

5.  6  straight  lines  versus  6  lines  with  five  electrode  shift  off 

We  predicted  that  subjects  would  not  be  able  to  reliably  discriminate  (d-prime  >1)  straight  lines  from 
shifted  lines  with  a  1-2  electrode  shift,  as  this  would  represent  spatial  distances  smaller  than  our 
previous  two-point  discrimination  threshold. 

All  subjects  reported  that  their  overall  impression  was  that  the  lines  were  continuous  and  straight.  As 
a  group,  when  electrode  shift  is  limited  to  1-2  electrode  offset,  subjects  were  not  able  to  detect  the 
linear  offset.  Moreover,  the  variability  as  seen  by  the  standard  error  bars  are  quite  small  indicating  that 
offsets  larger  than  our  assembly  error  do  not  affect  our  proposed  experiments.  In  general,  when  the 
offsets  were  larger,  group  perfonnance  improved,  however  there  was  more  inter-subject  variability  as 
seen  by  the  larger  error  bars.  This  may  be  due  to  individual  differences  or  due  to  the  few  trials 
executed  in  this  brief  experiment. 
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Group  data  is  graphically  presented  below  (Figure  1 1). 


Figure  11.  Vernier  Test 


Conclusions 

Given  d-prime>l  as  our  threshold,  it  is  clear  that  for  these  six  subjects,  no  offset  discrimination  was 
felt  for  1-2  electrode  offsets,  especially  given  the  tight  standard  error  bars.  These  results  confirm  our 
earlier  expectations  that  minor  geometric  errors  will  have  little  or  no  impact  on  our  proposed 
experiments  which  focus  on  gross  direction  and  speed  of  movement  (up/down/left/right).  Based  upon 
strategy  feedback,  the  best  performing  subject  was  able  to  discriminate  the  conditions  by  angular  offset 
rather  than  discontinuity.  For  future  reference,  it  appears  that  the  tongue  may  be  capable  of  small 
spatial  angular  discrimination,  which  may  be  important  for  some  applications  and  experiments. 
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Neuro-stimulation  optimization 

Three  subjects  participated  in  a  study  evaluating  dynamic  range  mapping  of  the  tongue  as  a  function  of 
scan  pattern.  Historically,  a  progressive  scanning  method  has  been  used  due  to  equipment  constraints. 
Changes  to  hardware  and  software  design  now  provide  the  ability  to  modify  the  electrode  activation 
sequence,  within  certain  safety  and  timing  constraints. 

Researchers  hypothesized  that  the  progressive  scan  pattern  may  inadvertently  be  inducing  spatial 
summation,  where  contiguous  electrodes  firing  together  feel  stronger  than  those  firing  individually. 
Therefore,  by  separating  the  electrode  firing  temporally,  perception  may  improve.  However,  it  is 
unknown  whether  the  microsecond  time-frame  scan  patterns  (beyond  known  temporal  gap  perception 
on  the  tongue)  in  this  experiment  will  alter  perception. 

In  this  experiment,  4  electrode  stimuli  (2x2  arrangement)  were  presented  randomly  across  25  regions 
across  the  tongue,  using  the  V100  electrode  array  (20x20  electrodes,  1.2  mm  spacing).  Subjects 
selected  a  minimum  perception  threshold  and  a  maximum  working  level  threshold,  defining  the 
dynamic  range.  The  system  allowed  a  maximum  of  19.5  volts  mapped  into  100  levels.  Four  scan 
patterns  were  used: 


Progressive  Electrodes  are  sequentially  addressed  and  activated  in  a  raster  scan 
pattern.  There  is  approximately  60  microseconds  between  adjacent 
electrode  firing.  The  adjacently  firing  electrodes  are  separated  by  1 ,2mm 

Random  All  electrodes  are  randomly  sequenced  for  addressing  and  activation.  A 
simple  pseudo-random  ordering  is  predefined,  and  once  defined  is  used 
during  all  sessions  (ie  the  random  sequence  of  firing  is  not  computed  for 
each  frame  or  run).  The  time  between  adjacent  electrode  firings  is 
between  60  microseconds  and  approximately  5.8  milliseconds. 

Adjacently  firing  electrodes  are  separated  by  1 ,2mm  to  24mm 


Interleave  A  A  predefined  electrode  addressing  sequence  is  generated  whereby  the 
array  is  raster-scanned,  but  two  electrodes  are  skipped  over  between 
sequential  firings.  Therefore,  adjacent  electrodes  will  have  at  least  360 
microseconds  between  firings.  Sequentially  firing  electrodes  are 
separated  by  at  least  3.6  mm 


Interleave  B  A  predefined  electrode  addressing  sequence  is  generated  so  that  there  is 
at  between  2.4mm  and  6mm  between  sequentially  firing  electrodes  and  at 
least  300  microseconds  between  adjacent  firings. 


In  any  case,  all  electrodes  are  activated  within  about  6  milliseconds. 

In  terms  of  minimum  threshold  perception,  there  were  no  significant  differences  between  the  scan 
patterns.  The  group  mean  was  37  units,  ranging  from  28-41.  In  common  with  previous  tongue 
mapping  data,  the  front  center  tip  of  the  tongue  was  the  most  sensitive,  with  sensitivity  dropping  in  the 
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posterior  and  lateral  directions.  Subjects  were  not  able  to  discern  any  explicit  difference  in  quality  of 
the  percept  in  this  study  between  scan  patterns. 


The  scan  patterns  differentiate  from  each  other  when  subjects  report  their  working  level  of  stimulation. 
Moreover,  the  working  levels  appear  to  divide  the  scan  patterns  into  two  groups:  lower  and  higher. 

The  Progressive  and  Random  scan  patterns  patterned  together  with  a  lower  working  level,  59  and  63 
stimulation  units  respectively.  The  Interleave-A  and  Interleave-B  patterned  together  with  a  greater 
working  level,  73  and  71  respectively.  Moreover,  this  pattern  was  evident  in  the  individual  data  as 
well  as  the  group  means.  This  grouping  pattern  extended  to  the  difference  between  working  level 
minus  minimum  threshold  level,  or  the  dynamic  range. 

These  results  suggest  that  scan  pattern  can  alter  perception  of  stimulation  intensity.  The  progressive 
and  random  stimulation  patterns  may  induce  spatial  summation  over  time.  As  a  result,  the  maximum 
working  level  is  overall  lower  than  when  a  small  temporal  offset  is  provided,  as  in  Interleave-A  and 
Interleave-B. 

There  are  a  few  implications  for  these  results.  First,  it  provides  evidence  that  micro-second  electrode 
stimulation  timing  can  alter  perception,  and  therefore,  neuro -stimulation  procedures  can  be  optimized 
for  a  particular  task  or  constraint.  Second,  micro-second  timing  patterns  may  not  be  explicitly  obvious 
to  the  perceiver.  Thus,  these  results  can  influence  the  design  parameters  for  a  particular  application. 
For  example,  if  there  is  a  design  constraint  to  keep  overall  voltage  as  low  as  possible,  one  would 
choose  the  Progressive/Random  stimulation  pattern.  However,  if  dynamic  range  were  important  to  the 
application,  one  may  implement  an  Interleave  scanning  pattern. 

Temporal  Perception  Enhancement  via  Pulsing  Scheme? 

In  reviewing  the  literature  on  temporal  perception  in  other  sensory  systems,  experimental 
methodologies  were  slightly  different  than  our  previously  reported  method.  Therefore,  in  order  to 
explore  whether  our  results  were  specific  to  our  methodology,  we  executed  a  comparative  study. 

In  the  previous  2AFC  Flicker  Fusion  experiment  (1mm  center-to-center  electrodes),  the  mean  of  16 
subjects  exceeded  threshold  gap  detection  (defined  as  >75%  correct)  at  temporal  gaps  of  75ms  or 
better.  The  -eontinuous”  condition  presented  a  repeating  scheme  of  two  25  ps  stimulating  pulses 
separated  by  a  lOOps  rest  period  repeating  at  1736Hz.  For  the  temporal  -gap”  conditions,  a  repeating 
pulse  scheme  of  stimulating  for  500ms  followed  by  no  stimulation  for  33-250ms,  depending  upon 
condition. 

In  that  study,  the  temporal  duty  cycle  for  the  gaps  was  not  50%  as  is  often  used  in  temporal 
discrimination  experiments  in  the  literature  for  other  sensory  systems.  To  better  compare  our  results 
with  the  literature,  we  repeated  that  experiment  with  five  new  subjects  for  comparison,  focusing  on 
just  the  fastest  temporal  conditions:  33,  50  and  75ms  using  a  50%  on/off  duty-cycle  scheme  using  the 
same  linear  electrode  array. 

As  before,  temporal  gap  discrimination  perfonnances  were  translated  into  a  percent  correct  score  for 
each  temporal  gap  and  subject.  Effective  discrimination  was  defined  as  a  group  performance  threshold 
greater  than  75%  correct. 
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Unlike  before,  this  small  group  (n=5)  did  not  exceed  performance  on  the  75ms  gap,  and  data  was  not 
collected  for  longer  gap  durations  (Figure  12). 
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Figure  12.  Flicker  Fusion 


75ms 


Percent  correct  perfonnance  trends  toward  better  performance  using  the  50%  duty  cycle  scheme,  but 
with  the  small  sample  size,  this  difference  is  not  yet  significant.  However,  an  ANOVA  (timing 
method  versus  duration)  revealed  a  similar  significant  main  effect  for  stimulus  spacing  (F(2,8)=4.473, 
p<.05).  As  the  temporal  gap  increased,  performance  improved,  regardless  of  timing  method.  This 
suggests  that  the  duty  cycle  timing,  at  least  for  the  new  subjects  surveyed,  did  not  significantly 
increase  performance. 

While  this  new  group  of  subjects  did  not  perfonn  above  threshold,  they  follow  the  same 
psychophysical  trend  as  before:  as  gap  duration  increases,  performance  improves.  Testing  with  the  16 
original  subjects  or  adding  additional  subjects  to  this  new  analysis  may  eliminate  this  difference.  For 
this  experiment  we  were  most  concerned  with  comparing  different  temporal  patterns  than  specifically 
replicating  the  older  study.  These  results  suggest  that  temporal  perception  may  be  a  generalized  skill 
that  is  minimally  affected  by  small  temporal  changes. 

In  the  finger-tip  tactile  sensory  literature,  temporal  discrimination  rates  are  faster  than  observed  in  our 
studies.  Based  upon  limited  observation  with  our  multi-sized  electrodes,  we  noticed  that  the  size  of 
the  electrode  appeared  to  be  related  to  better  temporal  gap  detection.  Recall  in  our  original 
experiment,  we  tried  to  run  the  flicker  fusion  experiment  on  the  254  micron  array,  but  our  pilot  data 
suggested  it  was  too  hard.  We  then  switched  to  the  1000  micron  array  and  observed  expected 
performance  patterns.  To  more  reliably  quantify  this  relationship,  in  the  next  reporting  period,  we  plan 
to  compare  flicker  fusion  performance  across  array  sizes. 
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2.6  Enhanced  Percept  -  Rifleman  Training 
Summary 

8  subjects  were  trained  to  hold  a  rifle  with  0  degrees  of  cant.  4  subjects,  in  the  control  group,  were 
given  verbal  training  and  4  subjects,  in  the  experimental  group,  were  given  automated  training  with 
feedback  from  an  integrated  BrainPort-based  tongue  display.  Both  groups  achieved  skills  at  holding 
the  rifle  in  a  stationary  vertical  orientation  and  performance  improved  with  training,  thus  proving  the 
hypothesis.  There  is  no  clear  difference  between  subjects  trained  by  human  interaction  or  those 
trained  exclusively  with  electro -tactual  stimulation. 

Method 

Preparation 

An  M-16  rifle  simulator  (G&G  Armament  GR16)  was  instrumented  to  detect  rifle  cant  (tilt/rotation) 
using  a  modified  BrainPort  VI 00  vision  system  (Wicab).  A  special  target  was  constructed  with  a  high 
contrast  horizon  located  12.5  cm  below  the  bull‘s-eye,  out  of  sight  of  the  shooter.  The  2  cm  bull‘s-eye 
was  placed  32  cm  from  the  floor.  The  target  was  mounted  approximately  3  meters  from  the  rifle  tip. 

The  VI 00  camera  was  mounted  to  the  front  sight  of  the  rifle,  out  of  sight  of  the  shooter,  and  software 
was  configured  to  detect  the  tilt  of  the  horizon  in  the  video  picture.  This  arrangement  provided  tilt 
accuracy  and  resolution  of  approximately  0.5  degrees. 

Software  was  further  configured  to  time  the  shots,  and  to  collect  and  store  shooter  data. 

In  preparation  for  each  subject,  the  rifle  was  configured  to  assign  unique  codes  to  the  data  files  of  the 
fonnat: 

nnnndtt — yyyymmdd-hhmmss.bpv 
nnnn  =  subject  ID 

d  =  trial  day  (4  =  final  test  data  on  day  3) 

tt  =  shot  number 

yyyymmdd  =  year  month  date 
hhmmss  =  hour  minute  second 

for  example:  1001101  -20090908- 1 4283 1  .bpv 

subject  1001,  day  1,  shot  1  on  8/9/2009  at  2:28:31  pm 

Additionally,  the  door  and  window  blinds  in  the  room  were  closed  an  all  lights  were  turned  on  to 
provide  a  quiet,  consistently  lit  environment. 


Subject  Orientation 

Upon  arrival,  each  subject  was  briefed  in  the  overall  requirements  for  the  study  and  provided  with  an 
infonned  consent  document. 

After  review  and  endorsement  of  the  infonned  consent  agreement,  the  subject  was  provided  with  a 
copy  of  Army  Rifle  Marksmanship  M16A1,  M16A2/3,  M16A4,  and  M4  Carbine  training  manual  and 
instructed  to  read  sections  4-5-a  and  4-5-b. 
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After  approximately  10  minutes,  the  trainer  reviewed  the  training  manual  infonnation  and  clarified  any 
difficult  to  understand  concepts.  Special  attention  was  given  to  rifle  cant  and  the  relationship  between 
bullet  drop  and  sight  alignment. 

The  subject  was  next  given  the  rifle  and  instructed  in  the  proper  manner  of  holding  it.  Following  that, 
the  subject  was  instructed  to  assume  a  comfortable  position  on  the  floor,  aiming  at  the  target  with  the 
right  hand  supported  by  a  sandbag.  Note:  all  subjects  were  right  handed  and  were  placed  in  the  same 
basic  position.  After  insuring  the  natural  point  of  aim  was  towards  the  target  and  that  the  subject  was 
comfortable,  instructions  were  presented  on  execution  of  the  training.  Finally,  the  subject  was  allowed 
to  draw  a  slip  of  paper  from  a  pile.  The  writing  on  the  slip  detennined  if  they  were  in  the  control  or 
the  experimental  group. 

If  the  subject  was  in  the  experimental  group,  a  final  5  minutes  of  instruction  were  given  on 
interpretation  of  the  stimulus  presented  to  the  tongue.  As  proof  of  understanding,  the  trainer  watched 
on  a  remote  PC  as  the  subject  was  instructed  to  rotate  CW,  rotate  CCW,  and  center  the  stimulus  and 
verified  appropriate  outputs. 

Baseline 

At  the  beginning  of  each  of  the  3  daily  sessions,  the  subject  was  asked  to  perfonn  10  shots  with  no 
guidance  and/or  stimulation. 

Training 

Following  baseline,  one  of  two  training  methods  was  employed  with  the  subject. 

Control  training  method 

Prior  to  each  sequence  of  10  shots,  the  subject  was  given  verbal  information  on  the  prior  10  shots.  For 
example: 

•  You  were  rotated  counter  clockwise  (CCW)  about  two  degrees  on  the  last  3  shots 

•  You  varied  +/-  1  degrees  around  zero  degrees 

•  You  started  2  degrees  CCW  and  ended  1  degree  CW 

•  AOK,  keep  doing  what  you  are  doing 


Following  the  verbal  instruction,  the  subject  was  asked  to  aim  at  the  target  while  the  instructor 
watched  the  PC.  Then  the  instructor  directed  the  subject  until  the  rifle  was  held  at  the  proper  perfectly 
vertical  orientation.  The  subject  was  told  to  relax  and  the  instructed  aiming  was  performed  one  more 
time. 

Following  the  training,  the  subject  was  instructed  to  take  10  shots.  Between  shots  the  rifle  software 
instructed  the  subject  to  lift  his  head  away  from  the  sight  for  approximately  5  seconds.  A  total  of  12 
training  /  shot  sequences  were  performed  across  3  days. 

Experimental  training  method 

Prior  to  each  sequence  of  10  shots,  the  subject  was  given  verbal  information  on  the  prior  10  shots.  For 
example: 
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•  AOK,  it  looks  like  the  hardware  is  functioning  properly 

•  Do  you  feel  you  understand  how  the  stimulation  guides  you  to  correct  your  aim? 

Following  the  verbal  instruction,  the  subject  was  asked  to  aim  at  the  target  while  the  instructor 
watched  the  PC.  Then  the  instructor  directed  the  subject  to  rotate  CW  and  then  rotate  CCW.  The 
purpose  was  to  insure  the  rifle  software  was  functioning  correctly.  The  stimulation  was  off  during  this 
task. 

Following  the  verbal  review,  the  subject  was  instructed  to  take  10  shots.  The  stimulation  level  was 
kept  at  a  constant  level  (recorded  in  the  log)  that  was  determined  by  the  subject  with  the  instructors 
guidance. 

Between  shots  the  rifle  software  instructed  the  subject  to  lift  his  head  away  from  the  sight  for 
approximately  5  seconds.  A  total  of  12  training  /  shot  sequences  were  performed  across  3  days. 

Final  evaluation  testing  method 

All  subjects  performed  3  groups  of  10  shots  following  the  third  day  of  training.  The  same  baseline 
instructions  were  used,  i.e.  no  feedback  or  other  infonnation  was  given  by  the  instructor  other  than 
j/ou  may  proceed1 .  Between  each  group  of  10  shots,  the  subject  was  instructed  to  stand  for  about  1 
minute  as  the  data  was  being  transferred  from  the  rifle  to  the  PC. 

All  in  all  18  series  of  10  shots  were  fired  by  each  subject. 

Day  1  -  Baseline  1,  Training  1,  Training  2,  Training  3,  Training  4 

Day  2  -  Baseline  2,  Training  5,  Training  6,  Training  7,  Training  8 

Day  3  -  Baseline  3,  Training  9,  Training  10,  Training  11,  Training  12,  Final  1,  Final  2,  Final  3 
Deviations 

The  protocol  was  executed  faithfully  with  a  few  notable  exceptions. 

•  Only  8  subjects  (4/4)  were  tested  as  opposed  to  10  subjects  (5/5) 

•  Failures  in  the  test  apparatus  resulted  in  the  loss  of  certain  data 

o  Subject  2,  samples  31-50 
o  Subj  ect  3 ,  samples  111-120 

o  Subject  8  used  a  cushion  on  day  3  for  health/comfort  reasons. 


Analysis 

An  ANOVA  statistical  analysis  was  performed  to  compare  the  effects  of  training  group  (BrainPort 
stimulation  versus  verbal  coaching)  on  performance  across  days  (days  1-4),  looking  at  average  tilt 
across  five  seconds  (Figure  13).  There  is  a  significant  main  effect  for  training  duration 
(F(3,18)=5.012,  p<.005),  where  subjects  made  significant  improvements,  reducing  the  tilt  of  the  rifle 
over  time  There  is  no  main  effect  contrasting  training  group  or  a  significant  interaction  suggesting  that 
both  training  paradigms  resulted  in  similar  endpoints.  In  other  words,  all  subjects  benefited  from 
either  training  protocol  reducing  the  overall  magnitude  of  tilt  within  four  days  training. 
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While  not  statistically  significant,  the  differences  between  the  two  group‘s  performances  appear  to 
diverge.  This  may  be  an  artifact  of  noise  or  could  reveal  differences  that  may  be  apparent  with  more 
subjects. 


Sharp  Shooter  Training 


■Stim 
■  Control 


Figure  13.  Training  Group  Performance 


Discussion 

These  results  suggest  that  providing  electrotactile  information  to  the  tongue  is  an  effective  alternative 
method  for  providing  rifle  position  training  to  a  novice.  Based  upon  this  small  study,  there  are  no 
differences  between  subjects  trained  by  human  interaction  or  those  trained  exclusively  with  electro- 
tactile  stimulation.  As  a  result,  training  techniques  could  be  modified  to  incorporate  aspects  of 
automated  training,  thus  directing  human  interaction  coaching  to  other  aspects  of  rifle  skill  acquisition. 
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3  HARDWARE  AND  SOFTWARE  DEVELOPMENTS 


Hardware  development  over  the  course  of  this  project  went  from  simple  linear  arrays  for  initial  spatial 
discrimination  testing  to  three  different  High  Density  Electrode  Arrays  (20x100,  40x50,  and  40x40) 
and  an  associated  simultaneous  control  system.  Later  stages  of  the  project  integrated  a  BrainPort 
vision  device  (BPV-V100),  allowing  real-time  image  flow  from  a  camera  or  other  source  to  be 
presented  on  the  tongue  via  a  400  electrode  array. 

3.1  High  Density  Array  Control  System 

Significant  engineering  effort  for  this  program  focused  on  building  the  High  Density  (HD)  Array 
Control  System,  used  for  discrimination  testing. 

The  overall  System  Architecture  is  shown  below,  Figure  14.  This  system  provides  micro-electrode 
transcutaneous  neuro-stimulation  on  a  new  scale:  thousands  of  simultaneous  active  electrodes  and  up 
to  96  unique  waveforms  (limited  only  by  device  memory). 

The  Experiment  Workstation  runs  the  experiment  control  software  application  and  provides  a 
Common  Software  Interface,  an  Ethernet  Connection  to  Control  Modules,  a  Socket  communications 
protocol.  It  is  designed  to  scale  with  the  control  hardware. 

The  HD  Controller  Boards  (Figure  15)  are  designed  to  manage  100  electrodes  each.  They  include  a 
Linux-based  Microcontroller  (Gumstix)  executing  custom  control  software  (the  Experiment  Virtual 
Machine),  an  FGPA  (10MHz  waveform  clock)  providing  Master-Slave  synchronization  across  all 
control  boards,  simultaneous  electrode  activation,  and  the  electrode  drivers.  The  control  boards  are 
designed  to  allow  each  electrode  to  be  an  active  electrode  (stimulating),  a  return  electrode,  or  floating 
(out  of  the  circuit). 


Figure  14.  High  Density  Array  Control  System  Architecture 
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TCP/IP  10/100 


HDA  with  2000  to  20,000 
electrodes 


Figure  15.  Core  Hardware  Architecture 


3.2  Hardware 

Significant  Accomplishments:  Design  and  manufacture  of  the  HD  Control  boards.  Each  board 
manages  the  state  of  100  electrodes  and  allows  simultaneous  activation  of  any  or  ah  at  a  given  time. 
Figure  16  shows  a  completed  HD  Control  board.  We  currently  have  22  boards  in  house. 
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Figure  16.  Complete  HD  Control  Board 


■  To  control  the  array  stack-up  described  below,  an  HD  Control  board  is  required  for  every  100 
electrodes.  Figure  17  shows  a  board  being  inserted  into  the  instrumentation  rack.  The  initial 
system  will  handle  2000  electrodes  (20  control  boards).  Each  board  has  a  cable  connecting  it 
to  a  set  of  electrodes  (cables  are  seen  hanging  off  the  front  of  several  boards). 


Figure  17.  Control  Board  Insertion 


■  When  fully  populated,  the  instrumentation  rack,  Figure  18,  contains  the  electro-mechanical 
interfaces  allowing  the  Experiment  Workstation  (a  PC  with  custom  software)  to  control  each 
HD  board  and  every  electrode. 
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Figure  18.  Populated  Instrumentation  Rack 


This  initial  implementation  was  sufficient  to  support  the  planned  experiments.  During  the  course  of  the 
project,  Wicab  and  DARPA  agreed  to  limit  testing  to  devices  with  2000  electrodes  before  deciding 
whether  fabrication  of  larger  capacity  arrays  is  necessary  (Note  that  the  system  is  designed  to  scale  up 
to  20,000+  electrodes). 

Wicab  used  this  high  density  array  system  to  conduct  studies  at  the  limits  of  spatial,  temporal  and 
contrast  discrimination.  In  addition,  the  control  architecture  supports  wavefonn  shaping  and 
simultaneous  activation  of  many  electrodes  (compared  to  previous  systems  where  only  one  electrode  at 
a  time  is  activated). 

High  Density  Array  Stack-up 

Figure  19  shows  a  100  element  strip  array  in  comparison  to  the  early  100  element  arrays.  Up  to  20 
strip  arrays  were  stacked  and  laminated  to  produce  an  array  assembly  used  for  testing.  Figure  (20. a) 
shows  the  completed  proof  of  concept  prototype  validating  the  design  and  fabrication  approach  to  the 
assembly. 
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Figure  20.  Strip  Array  Stack-up 

At  arms  length  the  individual  electrodes  are  barely  visible  (20. b)  to  the  naked  eye. 
Figure  21  shows  a  single  strip  array,  with  a  20  layer  stack  of  strips. 
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Figure  21.  Stackup  and  single  strip  array 


3.3  Software 

Software  modules  have  been  developed  for  the  experiment  system  throughout  the  TUNS  project  to 
support  psychophysical  experiments  and  provide  the  necessary  mechanisms  to  run  subjects  on  two 
tongue  stimulation  platforms  (WG-C200  and  HD  Controller).  Data  formats  were  also  developed  to 
encode  experiment  parameters,  wavefonn  parameters/timings,  stimulus  patterns,  etc.  for  use  with  the 
custom  software  being  developed.  The  data  formats  and  software  modules  designed  during  this 
project  are  summarized  in  this  document. 


3.4  Stimulation  Platform 

3.4.1  Virtual  Machine  Concept 

In  order  to  describe  stimulation  parameters  in  a  generic  enough  format  to  produce  similar  system 
behavior  across  hardware  implementations,  a  virtual  machine  design  has  been  used.  Tongue 
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stimulation  pattems/timings  are  parameterized  using  a  custom  instruction  set  that  executes  in  the 
virtual  machine  via  a  scheduled  run-to-completion  scheme.  Programs  written  using  the  instruction  set 
are  described  as  =Sthnulus  Programs4.  A  fixed  master  clock  drives  the  virtual  machine,  synchronizing 
voltage  outputs  at  the  electrodes.  Only  a  few  of  the  instructions  are  hardware  dependent  (those  related 
to  actually  commanding  the  hardware  voltage  outputs  and  wavefonn  selections)  implying  that  the 
virtual  machine  can  easily  be  ported  to  various  hardware  platforms.  Figure  22  shows  the  stimulaus 
program  workflow. 
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Figure  22.  Stimulus  Program  WorkFlow 


3. 4. 1.1  Instruction  Set 

The  instruction  set  is  outlined  in  detail  in  Appendix  A  -  Virtual  Machine  Instruction  Set.  Instructions 
have  been  designed  for  such  tasks  as  electrode  addressing,  voltage  adjustment,  waveform  selection, 
clock  adjustment  and  simple  execution  control  (looping,  conditionals,  etc.).  When  used  to  create  a 
Stimulus  Program,  execution  of  the  instructions  provides  fully  programmable  voltage  pulses  at  the 
electrode  array/IOD  limited  only  by  the  underlying  hardware  implementation. 

3.4.1.2  VM  State 

The  virtual  machine  state  is  outlined  in  detail  in  Appendix  B  -  Virtual  Machine  State  Definition.  In 
order  for  the  virtual  machine  to  execute  the  instructions  in  a  Stimulus  Program,  an  associated  virtual 
machine  state  is  maintained.  The  state  variables  are  referred  to  as  =registers‘  since  they  serve  the  same 
purpose  as  processor  registers  for  the  virtual  machine.  These  registers  are  broken  down  into  several 
subsets;  timing  registers,  loop  control  registers,  address  registers,  voltage  registers,  count  registers, 
block  voltage  registers,  loop  control  registers,  waveform  registers  and  hardware  specific  configuration 
registers.  Stimulus  Programs  can  use  these  registers  to  affect  how  the  voltage  at  an  electrode  is 
changed  over  time. 

3. 4. 1.3  Stimulus  Programs 

A  Stimulus  Program  defines  a  list  of  instructions  designed  to  produce  a  specific  stimulus  timing 
pattern  at  the  electrode  outputs.  Stimulus  Programs  define  exactly  how  the  electrode  outputs  will  be 
adjusted  over  time  based  on  values  stored  in  the  VM  State  registers  and  the  master  clock  rate.  The 
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stimulus  timing  and  voltage  output  features  are  limited  only  by  the  underlying  hardware  capabilities 
such  as  maximum  clock  speed  and  stimulation  power  supply. 


3. 4. 1.4  Hardware  Specific  Features 

For  each  platform  for  which  the  virtual  machine  is  implemented,  a  small  set  of  instructions  must  be 
implemented  uniquely  based  on  the  underlying  stimulation  hardware.  For  example,  a  WG-C200 
device  uses  a  single  DAC  channel  to  adjust  voltage  output  and  a  Multiplexer  to  select  the  single 
electrode  that  will  receive  the  voltage  output.  In  order  for  the  Stimulus  Program  to  adjust  voltage  on 
the  platform,  custom  code  must  be  implemented  to  communicate  with  the  DAC  and  Multiplexer.  The 
HD  Controller  voltage  output  cannot  be  adjusted  directly,  however  it  has  the  added  feature  of  custom 
waveform. 

3.4.2  HD  Controller 

In  order  to  support  high  density  arrays,  a  controller  was  designed  to  allow  parallel  boards  to  drive  a  set 
of  simultaneously-firing  electrodes,  synchronized  by  a  master  clock.  The  implementation  developed 
for  this  project  utilizes  a  Gumstix  ultra-mobile  single-board-computer  attached  to  a  custom-fab  board 
containing  an  FPGA,  SRAM  and  associated  drive  electronics  to  drive  100  electrode  outputs 
simultaneously  at  up  to  10MHz. 

The  boards  communicate  with  the  workstation,  acting  as  server,  via  100Mbit  Ethernet.  Software  on 
each  Gumstix  board  executes  a  client  application.  To  drive  electrodes,  each  board  has  a  100  pin 
interconnect  onto  which  a  ribbon  cable  can  be  attached.  On  the  other  side  of  the  ribbon  cable,  a 
number  of  different  array  configurations  can  be  connected  depending  on  the  requirements  of  the 
experiment.  For  small  linear  arrays,  a  subset  of  the  boards  may  be  used  to  drive  a  small  number  of 
electrodes  (e.g.  100  electrodes  can  be  driven  with  a  single  board).  For  higher  density  arrays,  any 
number  of  boards  up  to  a  total  of  20  on  the  currently  built  system  can  be  utilized  to  drive  up  to  2000 
electrodes. 

All  boards  are  linked  by  a  master  clock  bus  and  one  board  is  specified  as  the  master  to  drive  the  master 
clock.  The  FPGA  on  the  master  board  outputs  a  programmable  PWM  signal  to  all  other  FPGAs  on 
slave  boards  to  synchronize  electrode  outputs  across  all  boards.  Each  board  has  an  SRAM  directly 
attached  to  the  FPGA  to  store  waveform  data.  Waveform  data  consists  of  X/Y /Z  states  (X/Y/Z  states 
correspond  to  high/low/ground)  for  the  transistor  on  a  given  electrode  over  time  producing  a  — T- bit 
DAC”  style  output  on  the  electrode  when  stimulation  voltage  is  fixed.  The  internal  clock  on  the  FPGA 
provides  transistor  state  switching  at  up  to  10MHz  when  clocking  waveform  data  from  the  on-board 
SRAM. 

The  virtual  machine  implementation  on  the  Gumstix  boards  provides  methods  (via  WGIO  interface 
over  Ethernet)  to  program  the  FPGA,  load  waveform  data  into  the  SRAM,  set  the  master  clock 
frequency,  assign  waveform  selections  per  electrode  and  control  the  stimulation  power  relay  (on/off). 
The  software  on  the  Gumstix  also  provides  feedback  to  the  workstation  via  a  simple  web-based 
interface  (HTTP).  The  web  interface  provides  feedback  as  to  the  status  of  the  virtual  machine  and  has 
been  used  for  debugging  and  verification  that  the  intended  waveforms  are  being  sent  to  the  electrodes. 
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3.4.3  WG-C200 

To  support  early  experiment  execution  using  available  hardware,  custom  software/firmware  was 
developed  for  the  BrainPort  Balance  Device  C200.  A  device  executing  the  custom  software/firmware 
is  referred  to  as  WG-C200.  The  WG-C200  device  is  capable  of  stimulating  a  single  electrode  at  a  time 
when  used  with  discrete  IODs  (one  wire  per  electrode)  or  up  to  4  electrodes  at  a  time  when  used  with 
row/column  based  arrays.  Voltage  output  level  is  adjustable  by  a  DAC  and  the  active  electrode  is 
selected  using  a  multiplexer  (4  DAC  channels  are  used  on  row/column  arrays). 

The  WG-C200  device  communicates  using  the  WGIO  protocol  (see  Appendix  C  -  WGIO  Packet 
Definitions)  with  the  workstation  via  an  RS232  serial  cable  @  1 15200  baud.  Up  to  100  electrodes  can 
be  driven  when  connected  to  the  120  pin  connector  on  the  device  (for  discrete  arrays).  Row/column 
arrays  can  be  driven  with  a  row/column  adapter  and  an  appropriately  programmed  device  (supports 
both  18x18  and  25x25  row/column  based  arrays).  For  small  linear  arrays,  a  subset  of  the  100  electrode 
outputs  may  be  used  to  drive  a  small  number  of  electrodes  (e.g.  10  electrode  linear  arrays).  The  DAC 
can  be  adjusted  in  timing  increments  down  to  I  Ouscc  to  produce  a 

•  DAC  and  Wavefonn  Timing 

•  Virtual  Machine  Implementation 

3.5  Software  Modules 

The  software  modules  developed  for  the  experiment  system  can  be  broken  into  two  subsections,  pc- 
based  (workstation)  and  embedded  (HD  Controller).  The  windows  software  applications  provide  an 
interface  to  the  researcher  and/or  developer  for  executing  experiments  (Experiment  Controller)  and 
testing  system  operation  (HD  Diagnostic  Utility). 


3.6  Workstation  PC  Software  Modules 

3.6.1  Experiment  Controller 

The  Experiment  Controller  application  reads  configuration  from  a  Trial  Configuration  file  (see  the 
Data  Formats  section  for  more  details).  The  researcher  selects  an  experiment  (with  a  corresponding 
Trial  Configuration  file)  and  steps  through  the  software.  Experiment  specific  parameters  are  entered 
by  the  researcher  and  the  subject  is  presented  with  a  set  of  trials  for  which  they  respond  using  an  input 
device.  The  Experiment  Controller  software  records  the  subject  response  for  each  trial  and  generates 
an  experiment  report  upon  completion.  In  order  to  present  trials  to  the  subject,  the  Experiment 
Controller  communicates  with  the  HD  Controller  sending  it  trial-specific  parameters  which  determine 
the  stimulus  pattern  and  waveforms  used,  Figure  23.  Experiment  Controller  also  controls  logic  and 
stimulation  power  supplies  for  the  HD  Controller  via  RS232  serial  link. 
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Experiment  Controller  HDServer/Client  communication 
steps  during  execution  of  a  trial 


Figure  23.  HD  Server/Client  Communication 


3.6.2  Utilizing  the  VM  to  Present  Trials 

The  =Virtual  Machine1  running  on  the  HD  Controller  system  (see  the  Virtual  Machine  section  for  more 
details)  controls  communications  with  the  on-board  FPGA/SRAM  and  sequencing  of  electrode 
selections  to  the  attached  IOD.  This  is  accomplished  through  execution  of  a  Stimulus  Program  in  a 
scheduled  run-to-completion  scheme.  Stimulus  patterns  and  timings  are  defined  as  parameters  to  the 
Stimulus  Program  designed  for  a  particular  experiment.  For  each  trial  in  an  experiment,  the 
Experiment  Controller  sends  down  appropriate  stimulus  parameters  to  cause  the  Stimulus  Program  to 
present  the  intended  stimulus  to  the  IOD. 
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3.6.2. 1  Subject  Input  Devices 

Two  forms  of  subject  input  device  are  possible  with  the  Experiment  Controller;  the  hand-held 
controller  and  the  numeric  keypad.  The  Trial  Configuration  file  for  a  given  experiment  defines  which 
subject  input  device  is  to  be  used. 

3. 6.2. 2  Power  Supply  Control 

The  HD  system  includes  two  programmable  power  supplies;  the  logic  power  supply  (powers  logic 
circuits  on  the  HD  Controller  boards)  and  the  stimulation  power  supply  (powers  the  wavefonns  to  the 
IOD). 

3. 6.2. 3  Platfonn  Specific  Operations 

The  Experiment  Controller  contains  several  features  specific  to  the  WG-C200  platform  such  as  serial 
port  communications  and  use  of  J3AC‘  values  as  intensity  control.  It  also  includes  features  specific  to 
the  HD  Controller  system  such  as  power  supply  control  and  ramping  features,  support  for  waveform 
data  transfer  and  FPGA  file  support. 

3.6.3  HD  Diagnostic  Utility 

The  HD  Diagnostic  Utility  began  as  a  utility  to  visualize  Trial  Profiles  (see  Trial  Configuration  section 
for  more  details)  to  ensure  accuracy  when  developing  electrode  selection  patterns  on  high  density 
arrays  (up  to  2000  electrodes).  This  utility  provides  a  method  to  select  wavefonns  per  electrode  in  a 
similar  fashion  to  a  paint  program  and  allows  importing  of  images  to  create  a  Trial  Profile.  Other 
features  include  communication  with  HD  Controller  boards  and  functions  to  test  operations  such  as 
sending  waveform  data,  FPGA  data  or  Stimulus  Program  data  to  the  board(s). 


3.6.4  HD  Controller  Library  and  the  WGIO  Protocol 

The  HD  Controller  Library  implements  the  windows  side  of  the  WGIO  Communications  Protocol. 

This  protocol  provides  a  set  of  packet  definitions  allowing  Windows  applications  to  command  the  HD 
Controller  to  start/stop  stimulation,  update  the  Waveform  Definitions,  upload  an  FPGA  Program  file, 
update  Stimulus  Programs  and  assign  many  other  parameters  affecting  how  stimulation  is  presented  to 
the  IOD.  The  WGIO  protocol  provides  a  handle  into  the  state  machine  of  the  Virtual  Machine 
executing  on  the  HD  Controller  boards.  For  a  detailed  list  of  WGIO  packets,  see  Appendix  B  -  WGIO 
Protocol  Packet  Definitions.  The  HD  Controller  Library  provides  an  event-based  I/O  model  to  the 
parent  application  and  implements  a  socket -based  communications  layer  to  the  HD  Controller  boards 
via  Ethernet  acting  as  a  server  in  a  client-server  model. 
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3.7  HD  Controller  Board  Modules 


3.7.1  Gumstix  Platform 

Each  HD  Controller  board  requires  a  microprocessor  to  configure  the  FPGA  to  sequence  the  electrode 
outputs  and  communicate  with  the  PC  Workstation.  For  this  task,  a  =Gumstix‘  brand  single-board 
computer  has  been  chosen  for  its  speed,  size  and  versatility  in  communications  methods  (Ethernet, 
serial,  address/data  bus,  etc.).  The  board  includes  a  32-bit  Marvell  PXA270  processor  running  at  600 
MHz  with  128MB  SDRAM  and  64MB  flash.  The  board  runs  embedded  Finux  providing  TCP/IP 
communication  and  Compact  Flash  support  for  ease  of  development  and  integration.  The  FPGA  is 
connected  to  the  PXA270  via  its  Address/Data  bus  as  well  as  a  JTAG  interface  from  which  the 
Gumstix  board  programs  the  FPGA  during  the  boot  process. 


3. 7. 1.1  HD  Client  Application 

The  primary  software  application  running  on  the  Gumstix  platfonn  is  the  HD  Client  application.  This 
software  implements  the  virtual  machine,  programs  and  configures  the  FPGA  based  on  remote  TCP/IP 
communications  via  the  WGIO  protocol  and  provides  an  HTMF  based  status  web  page.  This  software 
maintains  communications  (as  client)  with  the  Experiment  Controller  (as  server)  on  the  PC.  As  an 
experiment  is  run,  the  Experiment  Controller  sends  commands  via  the  WGIO  protocol  to  the  HD 
Client  application  and  the  state  of  the  FPGA  is  updated  to  reflect  the  commanded  changes.  These 
changes  to  the  state  of  the  FPGA  are  moderated  by  the  virtual  machine  as  it  ticks  through  scheduled 
run-to-conrpletion  steps  of  a  selected  Stimulus  Program. 

3.7.  F2  Client  Diagnostic  Application 

The  Client  Diagnostic  application  was  developed  in  the  early  stages  of  testing  the  FPGA  to  provide  an 
easy  interface  to  changing  memory  locations  before  the  full  JTD  Client1  application  was  developed. 
Features  include  control  over  PWM  output  of  the  FPGA,  waveform  selection  per  electrode,  loading 
waveform  data  from  a  file  to  the  SRAM  attached  to  the  FPGA  and  FED  output  control.  This  utility  is 
mainly  useful  for  debugging/troubleshoot  HD  Controller  boards  exhibiting  unexpected  behavior. 

3. 7. 1.3  Virtual  Machine  Implementation 

The  implementation  of  the  virtual  machine  (VM)  on  the  Gumstix  platfonn  provides  implementations 
of  all  -general  purpose”  and  several  hardware  dependent  instructions.  Each  HD  Controller  board 
provides  a  trigger  input  which  triggers  execution  of  the  virtual  machine  -ticks”.  The  primary 
instructions  that  support  electrode  output  on  the  HD  system  are  related  to  wavefonn  selections  for  each 
electrode  via  the  Block  Voltage  registers,  Master  Clock/P WM  frequency  control  via  the  Tick  Unit 
register  and  control  of  starting/stopping  stimulus  programs  to  sequence  wavefonn  selections. 

3.7.  F4  Ethernet  Communications 

The  set  of  HD  Controller  boards  each  has  a  standard  Ethernet  jack  on  it  attached  to  the  Gumstix  board 
controlling  it.  These  Ethernet  jacks  are  connected  to  a  24  port  switch  to  form  a  local  network  along 
with  the  workstation  PC  and  the  hand-held  controller.  The  Experiment  Controller  (as  server) 
communicates  with  each  board  individually  (as  client)  via  TCP/IP.  The  Experiment  Controller  (as 
client)  communicates  with  the  hand-held  controller  via  TCP/IP. 
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3.7.1. 5  DHCP  server 

The  PC  Workstation  runs  a  simple  DHCP  server  that  supports  fixing  IP  address  to  MAC  address  for 
each  HD  controller  board.  The  subnet  used  for  the  local  communications  is  172. 16.5.0/255.255.255.0. 
The  hand-held  controller  address  is  fixed  at  172.16.5.1  and  the  PC  workstation  is  fixed  at  172.16.5.5. 
The  HD  Controller  boards  are  assigned  IP  addresses  sequentially  board  0  through  19  respectively 
numbered  172.16.5.10  through  172.16.5.29. 

3. 7. 1.6  PC  to  HD  Client  Connections 

Upon  booting  up  and  running  the  -HD  Client”  application,  each  HD  Controller  board  attempts  a 
TCP/IP  connection  (as  client)  to  the  Experiment  Controller  (as  server)  on  the  PC  workstation  on  TCP. 
Upon  connecting,  the  client  communicates  via  the  WGIO  protocol  and  waits  for  commands  from  the 
Experiment  controller. 

3. 7. 1.7  HHC  to  PC  Connection 

Upon  booting,  the  hand-held  controller  listens  (as  server)  for  connections  from  the  Experiment 
Controller  (as  client).  Once  connected,  the  hand-held  controller  sends  ASCII  fonnatted  text  packets 
reporting  the  status  of  the  slider,  knobs  and  buttons  as  states  change.  The  Experiment  Controller  can 
use  this  data  to  record  a  subjects  response  to  a  stimulus. 

3. 7. 1.8  Linux  Kernel  Module  and  Interrupts 

To  maintain  synchronized  operations  across  boards,  an  interrupt  is  triggered  with  each  tick  of  the 
Master  Clock/PWM  line. 

3. 7. 1.9  WGTrigger  Kernel  Module 

The  Master  Clock/PWM  line  is  used  to  synchronize  operations  across  HD  Controller  boards.  This  line 
triggers  both  the  FPGA  to  start  a  wavefonn  pulse  and  the  PXA270  on  the  Gumstix  board  via  a  GPIO 
pin  to  tick  the  VM.  A  Linux  kernel  module  (wgtrigger.ko)  has  been  developed  to  monitor  GPIO  22  to 
which  the  Master  Clock  line  is  connected  and  trigger  an  interrupt  on  the  rising  edge.  This  interrupt 
triggers  the  kernel  module  to  update  waveform  selections  in  the  FPGA  via  writes  to  the  Address/Data 
bus  and  signal  the  virtual  machine  to  execute  its  next  “tick”  in  its  currently  selected  Stimulus  Program. 

3.7.1.10  /proc  Filesystem 

The  wgtrigger.ko  kernel  module  utilizes  a  custom  file  /proc/wgtrigger  to  provide  a  pipe  for 
communication  between  user  and  kernel  space.  The  kernel  module  uses  it  to  signal  the  HD  Client 
application  to  execute  the  next  VM  tick.  The  HD  Client  application  uses  it  to  update  the  set  of 
waveform  selections  for  each  electrode  to  be  presented  on  the  next  Master  Clock/PWM  tick. 

3.7.2  Hardware  Interconnects 

3.7.2. 1  Address/Data  Bus 

The  FPGA  and  PXA270  are  connected  via  a  26  bit  address  and  32  bit  data  bus.  A  16  bit  Address/6xl6 
bit  data  bus  attaches  the  6  Static  RAM  (SRAM)  chips  to  the  FPGA  (See  TR-070054  for  more  details). 
Address  locations  have  been  defined  within  the  FPGA  for  various  features  corresponding  to  logic 
blocks  that  have  been  programmed  into  it  for  electrode  control,  LED  output,  Master  Clock/PWM 
generation  and  wavefonn  data  storage  to  the  SRAM. 
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3. 7.2. 2  JTAG  Interface 


A  JTAG  interface  (TDO/TDI/TCK/TMS)  is  used  to  program  the  FPGA.  In  order  for  the  PXA270  to 
program  the  FPGA,  4  GPIO  pins  have  been  tied  to  the  FPGA  programming  pins  and  a  customized 
version  of  Latticed  ispvmui  programming  tool  has  been  developed  for  the  Gumstix  platform. 

3. 7.2. 3  Electrode  Interface 

Each  HD  Controller  board  has  a  100  pin  connector  that  interfaces  to  a  100  conductor  ribbon  cable.  On 
the  opposite  side  of  the  ribbon  cables  is  an  adapter  providing  connection  to  either  a  10  electrode  linear 
array  (similar  to  that  used  in  the  WG-C200  device)  or  a  high-density  flex  array. 

3.7.3  Board-to-Board  Waveform  Synchronization 

Each  HD  Controller  board  supports  a  total  of  100  electrodes.  In  order  to  support  greater  than  100 
electrodes  in  parallel,  a  set  of  boards  uses  a  Master  Clock/P  WM  signal  to  synchronize  its  electrode 
outputs. 

3.7.4  Master  Clock/PWM  Signal  Output 

One  HD  Controller  within  the  set  is  configured  as  — Mster”  which  drives  its  Master  Clock/PWM 
signal  as  an  output  (all  other  boards  are  configured  as  — Sive”  which  disconnects  their  Master  Clock 
Signal  driver).  The  signal  frequency  and  duty  cycle  is  programmable,  typically  50%  duty  cycle 
running  at  <=  1kHz. 

3.7.4. 1  Responding  to  PWM  Signal  Input 

The  FPGA  and  PXA270  are  both  configured  to  respond  to  opposite  edges  of  the  Master  Clock/PWM 
signal.  The  FPGA  starts  firing  electrode  output  waveforms  on  a  falling  edge  of  this  signal.  The 
PXA270  triggers  a  -tick”  execution  in  its  VM  and  loads  up  wavefonn  selections  to  the  FPGA  in 
preparation  for  the  next  falling  edge  where  the  output  will  be  presented. 

3.7.5  Waveform  SRAM 

Wavefonns  are  generated  on  electrode  output  by  sequencing  the  -high”,  -low”  and  -ground” 
transistors  within  the  signal  path.  These  3  values  are  controlled  as  configuration  bits  that  can  be 
changed  at  a  rate  up  to  10MHz  by  the  FPGA.  To  toggle  these  lines  over  time,  the  FPGA  sequences 
through  a  series  of  addresses  on  the  attached  SRAM  modules  and  clocks  out  the  data  stored  in  the 
SRAM  directly  to  the  transistors  for  each  electrode. 

3.7.5. 1  Writing  to  SRAM  from  the  Gumstix 

The  SRAM  consists  of  6  individual  chips  with  16  bits  of  address/ 16  bits  of  data  64k  deep.  The  FPGA 
provides  an  address  window  to  the  Gumstix  for  writing/reading  to/from  the  SRAM.  From  the  Gumstix 
perspective,  the  SRAM  looks  like  a  contiguous  chunk  of  memory  as  seen  below,  Figure  24. 
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Support  for  127  Waveforms  in  SRAM 
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Figure  24.  SRAM  Memory  Map 


3. 7. 5. 2  Waveform  Selection  Multiplexer 

A  section  of  address  space  100  bytes  in  size  @  0x10100000  has  been  defined  on  the  FPGA  to  specify 
which  of  the  128  waveforms  in  the  waveform  table  should  be  output  on  each  electrode.  The  Gumstix 
writes  25  values  (32  bits  each)  to  change  the  waveform  that  is  presented  to  each  electrode. 

3. 7. 5. 3  Clocking  of  Waveforms  to  Transistors 

SRAM  data  starting  at  address  0x10000000  is  clocked  out  to  all  transistor  inputs  (X/Y/Z)  in  a  series  of 
4  clock  cycles  on  the  FPGA‘s  40MHz  clock.  On  a  falling  edge  of  the  Master  Clock/P WM  signal,  the 
FPGA  starts  its  waveform  output  sequence  sequentially  clocking  out  data  from  SRAM  to  the 
transistors  for  800 us.  The  data  in  the  SRAM  is  configured  by  the  Gumstix  via  the  address/data  bus 
before  the  Master  clock/PWM  signal  is  started.  By  default,  the  first  wavefonn  in  any  waveform  table 
should  be  programmed  to  select  the  -ground”  value  for  all  ticks  within  the  waveform  in  order  to  ensure 
that  the  system  can  always  disable  stimulation  by  selecting  wavefonn  0  for  all  electrodes. 

3. 7. 5. 3.1  Master  Waveform  Clock 

The  Master  Wavefonn  clock  runs  at  lOMhz  derived  from  the  40MHz  clock  rate  of  the  FPGA.  It  takes 
4  clock  cycles  to  clock  the  X,  Y  and  Z  states  out  to  all  100  electrodes  (100  x  3  bit  channels).  This  is 
derived  from  1  bit  for  X,  1  bit  for  Y,  1  bit  for  Z  needing  to  be  clocked  out  for  100  different  electrodes 
(3  x  100  bit  channels  written  out  128  bits  at  a  time  in  4  clock  cycles). 

3. 7. 5. 3. 2  Waveform  Data  Length  Limitations 

The  depth  of  the  SRAM  is  64K  which  detennines  the  maximum  length  of  a  wavefonn  that  can  be 
presented  to  the  electrodes.  The  configuration  chosen  provides  storage  for  up  to  128  unique 
waveforms  up  to  1.6ms  in  length.  This  is  derived  from  6  chips  x  16  bits  x  64K  depth  =  6291456  bits 
of  data  (786432  bytes)  divided  by  128  waveforms  divided  by  3  bits  per  tick  giving  16384  ticks  worth 
of  transistor  states  stored.  Since  the  wavefonn  clock  is  effectively  10MHz,  16384  ticks  will  take 
1.6384ms. 

3.7.6  Power  Supply  Control 

To  power  the  FID  Controller  system,  two  separate  power  supplies  are  used.  Each  supply  is  plugged 
into  a  hospital  grade  isolation  transformer  to  isolate  the  subject.  Each  power  supply  is  connected  to 
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the  PC  workstation  via  RS232  link.  This  RS232  connection  allows  the  Experiment  Controller  and 
other  utility  software  to  control  the  current  limits  and  voltage  level  for  the  supplies  remotely. 


3.7.6. 1  Logic  Power  Supply 

The  logic  power  supply  provides  a  constant  5.0V  to  the  HD  Controller  boards  to  power  the  Gumstix 
boards,  FPGA  and  related  hardware  via  on-board  regulators. 

3. 7. 6. 2  Stimulation  Power  Supply 

The  stimulation  power  supply  correlates  to  the  stimulation  level  that  is  output  at  the  IOD.  The  supply 
can  be  adjusted  during  an  experiment  to  find  a  level  that  is  perceived  as  -eomfortable  working  level” 
to  a  subject. 

3.7.7  DIP  Switches/LED  Indicators 

3.7.7. 1  FPGA  Programmed  Indicator 

The  blue  LED  turns  on  after  the  FPGA  has  been  successfully  programmed. 

3. 7. 7. 2  Stimulation  Active  Indicator 

The  amber  LED  turns  on  anytime  the  stimulation  relay  is  open  and  providing  stimulation. 

3. 7. 7. 3  Stimulation  Supply  Charged  Indicator 

The  yellow  LED  on  the  front  of  the  HD  Controller  board  indicates  when  the  capacitors  are  charged 
and  ready  to  stimulate.  This  LED  fades  as  the  capacitors  drain. 

3. 7. 7. 4  Master/Slave  Indicator 

The  red  LED  indicates  whether  an  HD  Controller  board  is  configured  as  a  master  or  slave.  If  a  board 
is  configured  as  master,  the  red  LED  will  be  on  and  its  Master  Clock/PWM  output  will  be  connected, 
otherwise  the  red  LED  will  be  off. 

3. 7. 7. 5  Master/Slave  Switch 

The  first  DIP  switch  is  used  to  set  whether  the  HD  Controller  board  will  be  configured  as  master  or 
slave.  If  the  board  is  used  on  its  own,  it  must  be  configured  as  master  and  a  jumper  must  be  installed 
in  the  jumper  block  to  complete  the  Master  Clock/PWM  loop-back  connection. 

3.7.8  FPGA 

3.7.8. 1  PWM  Signal  Programming 

The  Master  Clock/PWM  signal  is  programmable  from  the  Gumstix  at  address  0x10300000.  The 
PWM  signal  is  generated  using  two  counters  which  determine  the  -high  time”  and  the  -lew  time”  for 
the  signal  based  on  the  40MHz  base  clock.  This  provides  the  ability  to  program  the  PWM  output  with 
quite  high  resolution  in  the  required  frequency  range  for  the  system  (typically  <=  1kHz). 

3. 7. 8. 2  Waveform  Selection  Multiplexer 

The  FPGA  has  a  set  of  multiplexers  providing  the  ability  to  select  any  one  of  the  128  wavefonns 
loaded  into  the  SRAM  for  output  to  any  electrode.  This  implies  that  each  electrode  can  have  a 
completely  unique  waveform  or  can  be  programmed  to  have  the  same  wavefonn  as  other  electrodes 
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based  on  the  multiplexer  input  values.  To  specify  the  waveform  that  is  used  for  a  given  electrode,  an  8 
bit  value  must  be  written  to  the  corresponding  location  in  memory  location  0x10100000. 

3. 7. 8. 3  LED  Indicator  Controller 

The  FPGA  provides  access  to  changing  the  state  of  the  output  LEDs  as  bits  in  a  bitfield  at  location 
0x10200000. 


3.8  JTAG  Programming 

•  Lattice  semiconductor  provides  source  code  for  a  JTAG  programming  module  (ispvm  ui)  that 
could  be  customized  to  work  with  the  HD  Controller  platform.  Four  GPIO  pins  available  on 
the  Gumstix  board  were  programmed  as  I/O  and  the  ispvm  ui  tooPs  source  code  was 
configured  to  utilize  these  pins  for  TDI,  TDO,  TCK  and  TMS. 

•  The  HD  Client  application  implements  a  command  that  can  be  triggered  from  the  Experiment 
Controller  to  transfer  an  FPGA  program  in  the  form  of  a  .VME  file.  Once  the  .VME  file  is 
transferred  to  the  HD  Client  application,  the  ispvm  ui  command  line  utility  is  called  to  perform 
the  programming  step  placing  the  .VME  code  into  the  FPGA  via  the  JTAG  interface. 
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Deprecated  Software  Modules 

As  of  the  development  of  the  HD  Controller  system,  the  WGC200  platform  is  no  longer  used  to  collect 
data.  The  software  modules  developed  for  the  WGC200  are  frozen  and  will  be  considered 
=deprecated‘  as  we  do  not  plan  to  further  develop  the  software.  Software  modules  developed  for  the 
WGC200  platfonn  include: 


Software  Module 

Description 

WG-C200  Firmware 

Provides  a  subset  of  the  Wave  Generator  I/O  (WGIO) 
interface  enabling  experiments  to  be  developed  for  a 
modified  BrainPort  Balance. 

WG-C200  Simulator 

Simple  waveform  simulator  running  on  Windows  that 
outputs  electrode  stimulation  levels  graphically.  This 
application  was  no  longer  needed  once  the  WG-C200 
firmware  was  verified  to  be  working  properly. 

Array  Controller  API 
Test 

Simple  Windows  Forms  application  providing  a  graphical 
interface  to  the  WGIO  protocol  sending/receiving  packets 
to/from  the  WG-C200  device. 

Trial  Configurator 

This  tool  was  used  to  assist  in  creating  the  two  point 
discrimination  experiment  for  the  WG-C200  but  was  never 
finished. 

3.9  Data  Formats 

o  Trial  Configuration 

■  Stimulus  Program 

■  FPGA  Program  Files 

■  Waveform  Definitions 
o  Experiment  Controller  Report 
o  IOD  Definition  Files 

3.10  Large  Area  Array  and  V100  Software  Support 

Several  enhancements  were  made  to  the  TUNS  Experiment  Controller  and  related  software  in  support 
of  the  X16  experiment  system  (40x40  array)  and  features  for  wavefonn  optimization.  Additions 
include  interfacing  with  a  BrainPort  Vision  Device  V100  (BPV-V100)  to  display  real-time  images 
from  the  head-mounted  camera  to  the  40x40  intra-oral  device  (IOD)  array,  support  for  adjusting  the 
scan  pattern  on  20x20  IODs  (wavefonn  optimization  support)  and  vSight  enhancements  geared  toward 
TUNS  experiments  (feature  packs,  tongue  mapping,  etc.) 

3.11  X16  BPV-V100  Interface 

New  features  have  been  developed  for  the  TUNS  Experiment  Controller  software  allowing  the 
experimenter  to  connect  to  a  BPV-V100  device  during  a  trial.  When  enabled,  the  subject  wears  a 
VI 00  with  specially  configured  software  that  sends  the  camera  images  over  WiFi  for  presentation  to 
the  TUNS  HD  Array  Controller  (tower).  A  software  module  converts  the  optical  image  from  the  VI 00 
to  a  40x40  tactile  image  that  is  presented  to  the  user  via  the  X16  array.  The  user  interfaces  with  the 
standard  VI 00  controls  for  stimulation  intensity  control  on  the  40x40  array  attached  to  the  TUNS  HD 
array  controller  (tower).  This  enhancement  enables  experiments  based  on  visual  tasks  to  be  perfonned 

42 

Data  subject  to  restrictions  on  Cover  and  Notice  Page. 

Approved  for  public  release,  distribution  is  unlimited. 


using  the  40x40  electrode  tongue  array  and  allows  a  comparison  to  be  made  between  the  XI 6  (40x40) 
array  and  the  V100  intra-oral  device  (IOD)  which  only  has  20x20  electrodes. 

3.12  TUNS  vSight  Feature  Packs 

A  new  version  of  the  vSight  vision  display  software  has  been  released  as  part  of  the  BPV-V100 
product  that  provides  an  interface  to  developers  for  adding  custom  GUI  panels  at  run-time  in  support 
of  experiments.  Utilizing  this  feature  set,  custom  panels  have  been  designed  for  TUNS  experiments  to 
tailor  the  interface  for  tongue  mapping  and  waveform  optimization  tests. 

A  Wavefonn  Optimization  GUI  panel  has  been  developed  that  allows  control  of  a  special  load  of 
software  on  the  BPV-V100  device  to  vary  stimulation  pulse  parameters  at  runtime.  This  panel  allows 
the  researcher  to  provide  A/B  comparison  stimuli  to  a  subject  and  collect  data  from  the  subject  as 
needed. 

A  Tongue  Mapping  GUI  panel  has  been  developed  that  provides  an  interface  to  stimulation  of  fixed 
regions  of  the  tongue  for  a  min/working  style  experiment.  The  researcher  selects  a  region  of  the 
tongue  and  asks  the  subject  to  set  a  -minimum”  or  -working”  intensity  level  (depending  on  the 
experiment  parameters)  and  then  records  the  intensity  level  displayed  on  the  screen  to  — nip”  the 
tongue‘s  sensitivity  to  the  stimulus. 
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4  WAVEFORM  OPTIMIZATION 


Special  software  has  been  written  for  the  BPV-V100  device  to  provide  various  electrode  scan  patterns 
to  enable  testing  of  whether  waveform  scan  patterns  affect  perception  on  the  tongue  array.  These  scan 
patterns  include  standard  progressive  scanning  (adjacent  electrodes  are  pulsed  immediately  after  one 
another),  interleaved  scanning  (adjacent  electrodes  are  not  pulsed  immediately  after  one  another),  and 
random  scanning  (pseudorandom  electrode  scanning  pattern). 

A  simple  waveform  editor  has  also  been  developed  that  allows  creation  of  .XML  files  compatible  with 
TUNS  Experiment  Controller  to  drive  the  HD  Array  Controller  (TUNS  tower). 
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Appendix  A  -  Virtual  Machine  Instruction  Set 


Instruction 

Arguments 

Description 

AC 

Set  address  to  a  constant  value 

AR 

Set  address  to  a  register  value 

VC 

Set  voltage  to  a  constant  value 

VR 

Set  voltage  to  a  register  value 

RV 

Set  a  voltage  register  vavlue 

RA 

Set  an  address  register  vavlue 

TU 

Set  the  time  unit  register 

TR 

Set  the  tick  resolution  register 

LD 

Delay  a  constant  number  of  ticks 

LS 

Start  loop  with  a  constant  count 

LE 

End  loop,  decrement  count,  jump  to  start  if  count  >  0 

ET 

End  Tick 

RC 

Set  a  count  register  value 

LDR 

Delay  with  tick  count  loaded  from  count  register 

LSR 

Start  loop  with  count  loaded  from  count  register 

TRR 

Set  the  tick  resolution  from  a  count  register 

OIC 

Increment  a  COUNT[]  register  value 

ODC 

Decrement  a  COUNTQ  register  value 

ARC 

Set  address  from  ADDRESS[COUNT[x]] 

VRC 

Set  voltage  from  VOLTAGE[COUNT[x]] 

RSVD1 

No-op  (reserved) 

RSVD2 

No-op  (reserved) 

RSVD3 

No-op  (reserved) 

RSVD4 

No-op  (reserved) 

BNE 

Branch  if  COUNTjx]  !=  0 

BEQ 

Branch  if  COUNT[x]  ==  0 

ELSE 

Else  condition  for  if 

ENDIF 

Terminator  for  if 

RCC 

Copy  value  from  COUNT[bl  into  COUNT[aj 

RJMP 

Relative  jump  past  n  instructions 

ABI 

Address  increment  (block  mode) 

ABR 

Reset  address  (block  mode) 

VB 

Assign  channelVoltage[]  values  from  BiockVoitage[blocklndex][0- 
channelCount][tactor] 

BCC 

Assign  channelCount  from  a  constant 

BSC 

Assign  blockSize  from  a  constant 

BIC 

Assign  blocklndex  from  a  constant 

MODE 

Assign  array  mode  from  a  constant 

BBC 

Assign  blockBufferCount  from  a  constant 

ABAC 

Assign  absolute  address  in  block  mode  from  a  constant 

BSR 

Assign  blockSize  from  a  count  register 

BCR 

Assign  blockCount  from  a  register 

BBR 

Assign  blockBufferCount  from  a  count  register 

BIR 

Assign  blockBufferlndex  from  a  count  register 

BCP 

Assign  BlockVoltage[ARGO][][]  values  to 
VOLTAGE[BlockVoltageRef[ARG1  ][][]] 

BCPR 

Assign  BlockVoltage[COUNT[ARGO]][][]  values  to 

VOLT AGE[BlockVoltageRef[COUNT[ARG1  ]][][] 

VBR 

Assign  channelVoltagej]  values  from 

VOLTAGE[BlockVoltageRef[blocklndex][0-channelCount][tactor]] 

46 

Data  subject  to  restrictions  on  Cover  and  Notice  Page. 
Approved  for  public  release,  distribution  is  unlimited. 


WVUP 

Update  waveform  selections  from  BiockVoitage[bufidx][0][] 

PWM 

Start  PWM  with  freq=COUNT[ARGO]  duty=COUNT[ARG11 

WVTS 

Activate  a  waveform  table  from  the  VM  table  list  (send  it  to  the 
FPGA) 

BNEO 

Branch  ARG2  steps  if  COUNT[ARGO+COUNT[X]]  !=  0 

ODCO 

Decrement  a  COUNT[ARGO+COUNT[ARG1]]  register  value 

BVR 

Set  a  BlockVoltageRef[ARGO][ARG1][COUNT[ARG2]]  register  to 
ARG3 

RCO 

Set  COUNT[ARGO+COUNT[ARG1]1  register  to  ARG2 

RCCO 

Set  COUNT[ARGO+COUNT[ARG1]]  = 
COUNT[ARG2+COUNT[ARG3H 

BEQO 

Branch  ARG2  steps  if  COUNT[ARGO+COUNT[X]]  ==  0 

BVRR 

Set  blockVoltageRef[ARGO][ARG1][COUNT[ARG2]]  to 
COUNT[ARG3j  +  COUNT[ARG4j 
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Appendix  B  -  Virtual  Machine  State  Definition 


Data  Type 

Name 

Description 

UINT32 

pc 

Program  counter  (in  bytes) 

UINT32 

Ic 

Loop  counter  (current  index  into  loopcount,  increments 
when  a  new  loop  is  started,  decrements  when  a  loop 
completes) 

UINT32 

tu 

Time  unit  (defaults  to  microseconds) 

UINT32 

tr 

Tick  resolution  (#  of  ticks  in  „tu’  units  between  VM  ticks) 

UINT32 

et 

End  tick  (ends  a  run-to-completion  series) 

UINT32 

dr 

Delay  register  (holds  #  of  ticks  to  delay) 

UINT32[] 

loopStart 

Stores  pc  when  a  loop  starts 

UINT32[] 

loopCount 

Loop  counts  (dimension  determines  max  #  of  nested  loops) 

UINT32[] 

regV 

Voltage  registers 

UINT32H 

regA 

Address  registers 

UINT32[] 

regC 

Count  registers 

UINT32 

ainc 

0  -  do  not  increment  address,  1  -  increment  address  (used 
for  row/column  IOD  block  mode) 

UINT32 

arst 

Address  reset  (0  do  nothing,  1  reset  tactor  address  to  0  on 
next  tick) 

UINT32 

dacval 

Current  DAC  value 

UINT32 

tactor 

Current  tactor  address 

UINT32 

arrayMode 

Current  array  configuration  mode 

0:  Discrete  tactor  scanning  mode 

1 :  Block  tactor  scanning  mode 

2:  Discrete  parallel  mode 

3:  Block  parallel  mode 

UINT32 

addressMode 

Current  tactor  addressing  mode 

UINT32 

voltageMode 

Current  tactor  voltage  mode 

0:  Single  DAC 

1:  Multiple  DAC 

2:  Simultaneous 

UINT32 

blockSize 

Number  of  factors  in  a  block  for  block  addressing  mode 

UINT32 

blockBufferCount 

Number  of  block  voltage  buffers  to  allocate  (corresponds  to 
number  of  full  “frames”  of  waveform  selections  for  each 
tactor) 

UINT32 

blockBufferlndex 

Index  into  the  blockVoltage  array  indicating  the  currently 
selected  block  voltage  buffer  to  present 

UINT32 

channelCount 

Number  of  channels/blocks  for  multi-DAC  mode 

UINT32[] 

channelVoltage 

Current  DAC  value  for  each  channel  in  multi-DAC  mode 

UINT32[][][] 

blockVoltage 

Array  of  values  sized  by 

[blockBufferCount][channelCount][blockSize]  used  to 
indicate  which  voltages  should  be  applied  to  each  tactor 
and/or  which  waveforms  should  be  applied  to  each  tactor 

UINT32[] 

blockVoltageRef 

Array  of  indices  into  regV[]  that  can  be  used  to  copy  regV[] 
values  into  a  blockVoltage[][][]  buffer 

UINT32 

fpgaFileCount 

Number  of  FPGA  files  (.VME)  loaded 

CHAR[][] 

fpgaFilename 

Filenames  of  .VME  files  loaded 

UINT32 

waveformT  ableCount 

UINT32 

waveformT  ableCount 

Number  of  waveform  tables  that  are  loaded 

U 1 N  T  8  []  [] 

waveform  DataTable 

Points  to  dynamically  allocated  waveform  table  arrays 

UINT32[] 

waveformCounts 

Number  of  waveforms  in  each  table 

UINT32 

waveformSizes 

Sizes  of  each  waveform  in  bytes 
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Appendix  C  -  WGIO  Packet  Definitions 


Command 

Arguments 

Description 

RDP 

Return  Data  Packet 

SPLS 

List  loaded  Stimulus  Programs 

SPSET 

Set  the  current  Stimulus  Program 

SPLD 

Load  a  Stimulus  Program 

SPDEL 

Delete  a  Stimulus  Program 

RASET 

Set  one  or  more  address  registers 

RVSET 

Set  one  or  more  voltage  registers 

RCSET 

Set  one  or  more  count  registers 

SPSTAT 

Get  device  status  packet 

SPGO 

Start/Resume  the  VM 

SPHALT 

Stop/Pause  the  VM 

PINGCFG 

Configure  PING  packets 

PING 

Ping  packet 

RESET 

Reset  the  VM 

CLEARSP 

Delete  all  Stimulus  Programs 

RAGET 

Get  the  value  of  an  address  register 

RVGET 

Get  the  value  of  a  voltage  register 

RCGET 

Get  the  value  of  a  count  register 

RBSET 

Set  one  or  more  block  voltage  buffer  registers  to  a  value 

BREFSET 

Set  one  or  more  BlockVoltageRefnnH  registers  to  a  value 

RBGET 

Request  a  packet  containing  the  value  of  a  block  voltage 
buffer  value 

CCONFIG 

Client  configuration  packet 

VMEFILE 

FPGA  .VME  file  packet 

WAVETBL 

Waveform  data  table  packet 

CLRDY 

Client  ready  packet 

RBREF 

Set  one  or  more  values  in  BlockVoltageRef[]  to  a  value 
from  the  VOLTAGEQ  array 
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LIST  OF  ABBREVIATIONS 


AFRL  Air  Force  Research  Laboratory 
API  Application  Program  Interface 

DARPA  Defense  Advanced  Research  Projects  Agency 

HD  High  Density 

HDA  High  Density  Array 

IOD  Intra-Oral  Device 

IRB  Institutional  Review  Board 

Electrode  Electrode  on  stimulation  array,  comparable  to  pixel  on  visual  display 
TUNS  Tactical  Underwater  Navigation  Systems 
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